text:
InfoQ Homepage Presentations Scaling Patterns for Netflix's Edge

Scaling Patterns for Netflix's Edge

Justin Ryan talks about Netflix’ scalability issues and some of the ways they addressed it. He shares successes they’ve had from unintuitively partitioning computation into multiple services to get better runtime characteristics. He introduces us to useful probabilistic data structures, innovative bi-directional data passing, open-source projects available from Netflix that make this all possible.

Justin Ryan is Playback Edge Engineering at Netflix. He works on some of the most critical services at Netflix, specifically focusing on user and device authentication. Years of building developer tools has also given him a healthy set of opinions on developer productivity.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Ryan: My talks can be a little less dense on the slides here. We're going to bring it down a little. I guess we're going to talk about laundry. Everyone knows how to do laundry, we all know how it works. We get a big pile of clothes, are all over the place, and you have to start folding them and you have different clothes. I got a family of four, so I have my wife's clothes, her pants. I got kids' socks all over the place and I need a way of folding it. I have a certain way of folding it, and then my spouse has a certain way of folding it. The way she goes about it is she folds, which is great - it's great that she's folding laundry, and I don't have to do it - but what she's going to do is stack it all up in a big pile. Every time you get a piece of laundry and you're folding it, she just puts it into a pile, and then on goes the next one, and you end up with a pile of some kid pants and my shirt, and to put it all over the house, you have to walk over the house. Admittedly, I do appreciate the exercise, can't argue with that, but parents, we're busy - we don't have time for that. I have a slightly different way of doing it. I take over the whole living room - I'm talking the couch, the coffee table, the ottoman. I make separate piles for each kind of clothing - kid's pants, kid number one pants, kid number two pants, something like that. I take over the whole living room, then I can go around the house putting my pants away, my shirts away, my kids' pants away, kids' shirts away, and it goes really fast.

What we're doing here is making a space-time compromise. It sounds like software, but it's not. It's t-shirts and pants and stuff like that. I was able to take the same amount of time folding. It didn't take me crazy long to fold the clothes. I just took them in more space, and then when I was able to put them away, I was able to save time. That's this classic trade-off that we as architects have to think about. What happens when you have 158 million t-shirts?

Last reported by Netflix, we have 158 million memberships with the service. When I started eight years ago, we had less than a million, so at least you can get a sense of this growth and what happens over time and how things aren't the same as they used to be. It is not one monolith server we run in one little data center. It is the full-blown microservice architecture with tons of features, with services calling services, almost like to say ad infinitum because it does feel like that sometimes. That's a lot more complex than a few servers. There's a lot of servers, and there's problems that come up when you have that many servers and that many systems running altogether, as you can imagine.

You might hear me say edge. Hopefully, you read the title of the talk - edge is in there. That means the edge of this graph. For most people, the far left here where it says edge. I'm talking about these four nodes over here, the edge when the data comes into the service. There's a lot of other services going on here. I'm just not generally talking about that. I'm talking about edge. It's where I work, and it's where we saw the most scale.

Scaling Problems

When we talk about that scale, there's problems. That's what I was alluding to, and I also want to get to a little bit. Like, debuggability. When you have 1,000 servers, you have 1,000 logs that go through, and there's no way you're going to go to every single machine and look at the logs anymore. That's just out, that's not an option anymore. Debugging these problems becomes harder because a lot of problems can lurk in 1,000 servers.

Infrastructure - when you can keep some piece of infrastructure in one box and tracks everything, that's great, but as these number of servers is growing, you deal with 10,000 instances, 20,000 instances, it gets more difficult. Things start to fall apart. Then the third point there is managing. We subscribe to DevOps. I'm the developer, I'm operating, and I'm deploying it. I now have to deploy 1,000 instances. Now, there's automation around it, but things just take longer now, and I have to monitor more servers. Things just get harder.

On the topic of scalability, it's the ability to take more traffic without noticeable degradation. I should be able to take more traffic and not have the whole system collapse. It means that I have problems that don't exist on a smaller system, but it also means that I want to give myself some headroom. I don't need to design for 10X scale, I just want 2X. I want to be able to failover to a region. I want some new feature to ship and not have the system fall over.

That's where I want to talk about scalability and where I'm coming to here. I'm trying to give myself a little leeway here as my system grows because, as you saw, the system is going to grow. That's a given. The way we do that, we make some trade-offs. We're architects here, this is what we do. We look for a degree of freedom that we can move around in to make the solution work without either costing lots of money or lots of servers, or something like that. A classic trade-off here would be the CAP theorem. In Netflix, we have a service discovery service called Eureka. It's open-source, feel free to check it out. It leans heavily on the A and the P, available and partition tolerant. Then when it does that, it takes all the instances it's keeping track of, and it distributes it out over to all the other instances, meaning if you get a partition, it still knows about these instances in general. That's a trade-off. It also means that when you have 10,000 instances, and you add one more, you have to go tell the other 10,000 about that one new one, and that gets into the infrastructure side of things.

I'm going to focus on the architectural things that we can do as architects and not on the policy decisions. If you have a policy to keep data for six months and you turn it to three months, that's less data, that will cost you less money, go for it, save us money. That's not what I'm talking about. That's a turning the dial kind of thing, like just turning the dial down. Not super helpful. Not what I'm talking about, I should say. Likewise, if you run less servers, there'll be less problems. Don't run less servers just to run less servers. Don't run them hot. We've learned that this has caused additional latency, and that's not great either. I'm talking about being smart about this complexity as we add it.

The things that I'm going to show, I have some case stories I'm going to go through. Each of them was implemented with one or two engineers over the course of a few weeks, maybe a month. The number of person-months here is pretty low. I would say the total amount of work is not a lot. It's something that I think you guys could walk away with a little bit. I would say that all the technologies we used was generally off-the-shelf. This is not some academic paper you have to implement. You should feel perfectly capable of implementing the things that I'm going to show here.

I'm going to go through five use cases today. I'm going to focus a little more at the beginning, and as I go on, they're going to get a little shorter and faster just so I can get through enough material. First one, how do you tell someone is logged in? How do you tell if they're a member? That's pretty important. You're going to show a very different website if someone is logged in or not or if their membership is active or not. And you have to tell this on every request, this is a stateless system. They're going to come with a request, and another request. You have to know if they're a member or not. You could say, "Every request comes in the system, we just go to the database. Look it up. No big deal." It does become a big deal when you're doing 2 million requests per second. That is the scale that I'm talking about here. There are databases that scale like that. I would like to avoid it. My scalability message is that I want to give myself a lot more leeway. If I don't have to put pressure on the database, let's not.

Let's go back to what we have. We have cookies, that's how most of our devices talk to Netflix or service, that gives each the cookies. These are long-lived cookies. They sit on your box for a year, nine months, but in there is a little expiration date that we put in, and this expiration date so tells us how long we can trust the cookie. If it comes into our system and we crack it open, and we look, we can look for its expiration. That's the dot in the middle that you see up here. If it has not expired, we're just going to trust it. This fits our business use case. At Netflix, if you're familiar with it, you're going to load it up, and maybe the last time you used the service was last night, so it's been eight hours. We're going to need to go make sure you're still a member. After that, we can give you a new cookie that's good for eight hours. For the rest of your TV watching binging for that night, we can trust that cookie, and that fits our use case. If you're a member, at the beginning of the night, you're probably a member at the end of the night. Works great.

Let me just walk through how that works because we're going to turn it up a notch later. What we have here is a layer 7 proxy. This would be equivalent to an ALB, Nginx, something like that. In our case, we're using Zuul, it's another open-source project from Netflix that you can use. It lets us put custom logic in there. If you see me putting code in this box and you can't, or you have to write in Lua or something, that's what's going on here. We're doing this in Zuul. Cookies come in, we crack it open, and we see if it's expired. If it's not expired, let it through. If it is expired, we're going to go to the server account service. Just look up, hit the database, see if they're still a member. It's very straightforward. Either way, we're going to send it down, and once we have this refreshed answer, if they're a member or not, we actually send out a new cookie that's good for eight hours, and that buys us some time.

The thing is that wasn't good enough. A product manager will come back, say, "No, we want to log someone out immediately." There's a lot of different use case of this. You might be on the phone as customer service, saying, "I see some weird activity. I just want everyone logged out. I don't want any activity on this." Maybe you've given your account to your college kid and they're watching some risque stuff, and you just don't want to know about it. "Sign them out. I don't want it. I want it gone right away."

The business requirement now is we have to log them out instantaneously, but we don't want to hit the database. That's what we're trying to save here. We came with a system called Melnitz, and it's focused on that first eight hours. What I have to do now is identify who has had activity on their account in that first eight hours and let the expiration thing take care of itself. That's done, we figured that out. Let's focus on that first eight hours, and your set of customer IDs of who's changed their account in the last eight hours. 150 million members, how many have really hit the sign out of all devices button or called customer service? It's a small number. Here's our set.

The thing is, that isn't that small, and I have to get it out. There's two pieces to this. One, I'm going to get the set, and I'm going to send it out to my layer 7 proxies. Let's talk about the size here. This goes a little bit to the scalability thing. If I just do this at a short cadence, four requests per second, that's still 115,000 entries I need to keep track of. It might be more, it might be less, depending on time of day, stuff like that. That's a lot of data to ship out to every single one of our layer 7 proxies, there's thousands of them. I don't want to ship that much data, and I don't want to do it every minute, which is what we're going to be doing.

We introduced a Bloom filter. If you haven't heard of it, that's ok. Let me talk about why it's ok. Actually, what it gives us is it going to answer this question. Is something possibly in a set, the high degree of probability? Is it a bad interview question what a Bloom filter is? Yes. You shouldn't have to know about it. Does it feel like it's out of an academic paper? Yes, it does, because it's shown like this. Is it available in Wikipedia? Yes. You can read about it, and that's where I want you to go for more details. I'm not going to get too much into it. It's also sitting in a Guava library, so if you're a Java developer, it's sitting in Guava and you can use it. It's in 10 different libraries, and I promise you it's in your language of choice. It's easy to use.

What it's giving us is we can take this set of IDs, X, Y, Z, and we splay it all over this byte array, and this byte array is going to take up a lot less space than those individual IDs. There's going to be some places where they conflict. That's ok, it's going to happen with a certain probability that we can tune, and we turn it way down or way up, whatever works.

Let's talk about how this works in here. Same layer 7 proxy, same expired logic, but now if you're not expired, we're going to send you through this Bloom filter, and the way the Bloom filter works is, it's a set. You say, "Is this thing in a set?" That's the only thing we're trying to answer. It says no or maybe. In the no case, just flow it through, and that is the 99.99% of all requests. For that small 0.01% of the time, we get to maybe answer, we go to the account service, we look it up. This means we're actually hitting the database a little bit more, but not much. This is very reasonable.

I'm going to talk about how I set it up. I'm not going to go into as much detail on the other use cases, but I think this is really relevant because I want to show how accessible this is. We had an account service, that already existed. This doesn't work without an account service. It was already sending events out over some Kafka-like stream this - this already existed for us. We wrote a service that just listened to events and then saved in the database. If things go right, this is like five lines of code. This is pretty simple, straight forward. We then read from this database every one minute. Say, "Give me all the entries for the last eight hours," we put that into the Bloom filter, and we ship it out. We ship it out using a PubSub system. We have one internally, it's not open-sourced, but there are other ones available. Apache has one that came out of Yahoo, which is very similar to the one that we have. Redis has a PubSub, Google offers one, PubSub is generally available. We just throw it out there, here's the new list of all the customer IDs in the last eight hours who might have changed their account, and now it's on our layer 7 proxy.

We now have this near real-time data set we can look at to log people out. We haven't increased the amount of calls to the database, so that was one of our goals here. We're pretty happy with it, but we did have to make some trade-offs. One, there's now a probabilistic data structure in there that I trust you, every new employee that joins the team is going to have to go to Wikipedia and read up on it. It is a library, you do get to call, but you should understand what's going on. We traded a little more code complexity to get this neat space savings.

Also, off-the-shelf components. We could have sent those events from the OC service, the one that's doing this account stuff straight to the layer 7 servers, those proxies. That would work, that would be like instantaneous. There's a lot of problems that could happen there, and we can handcraft it, but we can also just use off-the-shelf components. I use off-the-shelf Kafka, and PubSub and I wrote a service that's like five lines of code or two services that's five lines of code. That was worth it. I was much more willing to have two simple services than try and take my account service, which is super important, with my layer 7 proxy, which is super important, and put a whole bunch of complicated code in there. That's the trade-off I made here. That was the first use case. I'm going to go through four more.

We're going to talk about the needle in the haystack kind of problem. I talked about how, when you have 1,000 servers, and you're trying to find a log in there, how do you find it? How do you find this problem? We need to solve this all the time. This is just a classic debugging kind of thing. You've got to find an error message. If I do this at the pre-stated 2 million requests per second, and let's just say every request for some metadata is about three kilobytes - I showed you that graph, they're going to call other microservices, and we want to record all those. You multiply that out, that's 4.6 petabytes a day. You could do that, you can put that in Elasticsearch. I trust you there's a large company out there who is willing to take your check and store all this. I'd like to avoid that.

Let me just start with the basic architecture - the pitchers. You have a service, it's publishing all these events, all these logs, and let's say we put into Elasticsearch, you can search for it. For those who can't see in the back, there's a thick red line between the service and Elasticsearch and a thin line to let's say Kibana or your UI over it, but you have to load all this stuff into Elasticsearch, and I don't want to put that 5.6 petabytes in there.

In comes Mantis, a recently open-sourced tool from Netflix that we've been using internally for a while, and I absolutely love it, and I think you will too afterwards. It is event stream processing. It is not specific to logs. It's there for events, and it's going to give us two big constructs. It is going to put an agent in your service. This is a local agent that you run that you give it the event and it is responsible for getting that event out of your box, so think of it as for every request that comes in, we create an event, that would be our equivalent of a log, and it's going to send it to a Mantis source job. It has a cluster that's dedicated to receiving these events and sending it on to somewhere else. I've drawn it with really thick lines because I haven't told you anything else about why this makes it special.

The thing that makes it special is that Mantis has a query language. MQL - Mantas query language, pretty obvious. You can write this query, you can say, "Show me all the people in Brazil watching Episode 3 of "Stranger Things" on an iPad. Super specific, that's a select statement right there with some where clauses. That will get loaded into the source job, and it's only going to allow things that match that filter to go out. Now we're talking about what's going to Elasticsearch. It's a lot smaller. For the people in the back, there's a thin red line now going to Elasticsearch. There was a trickle of data. Only those people in Brazil on an iPad watching Episode 3 of "Stranger Things."

What gets a little bit better at Mantis is it takes that MQL and pumps it down to all the agents. You might have 1,000 servers, 10,000 servers, or 5. That query is going to go down to the agent. He's going to run the pre-filter and only send off box those messages, and that is a trickle of events. I can get extremely specific tracking. I can see these requests come in live. They have a browser plugin where you can watch this stream. I'm getting it live, no huge overhead. I'm not storing tons of data here. This is what makes Mantis a big win for us.

The thing that it does is avoid work. We're not shipping all this data off box. We had to put an agent in our box, but we avoid a lot of network traffic going on here. I'd say those are the two pieces here. I'll start with scales from zero source solution here. I could have this thing on all night, it's only going to match those iPads in Brazil, that's awesome. I could leave it on all month, no big deal. I could go home at night-time and turn everything off, and there's no additional impact to the system. The trade-off here is, I don't have long-term historical data here, but I do get a really easy way of starting up a new query. I don't care about the cost because it's going to be so optimally efficient to what I am looking for. That's the trade-off.

The other trade-off is this Mantis query language. I had to put an agent in my box, there's something in my runtime now. They control it, if I want to get more complicated and it's not part of the query language, I can't do it, but this is something where I'm willing to hand it off to them. I trust this team to put code in my box. I trust this team to enhance this query language over time. I could have put custom code in my box, but I couldn't put that custom code in all the boxes, and so I'm willing to make the trade-off.

That was two case stories, let's talk about the third one. This one is a little specific to when you have extremely common data for your requests. Let me explain that a little bit. Membership, that's a common one, or user preference. Let's say they are a premium member, and you will always give them a little more data on every request. Every request comes in your box, and based on the membership, maybe you do something different. Something for Netflix in equivalent might be membership levels. There's a two-stream and a four-stream. Knowing if it's a one-stream or four-stream helps us change our behavior a little bit. This is important data for us, but you might have more important data for a user preference or something like that. Super common data.

You can imagine how you load it. A request comes to the system, gets loaded into API, he requests the service plan from the database. Call goes to mid-tier, he requests it. Call goes to mid-tier B, he requests it. Goes to mid-tier C, she requests it. We got a bunch of requests coming out of the database. One request in, four requests to the database. This is that kind of scalability thing I'm trying to avoid a little bit. Let me layer in one thing. We have this thing called passports, we introduced this for identity. It tells you who this call is made on behalf of, and we do this right at this first layer 7 proxy. We figure out who is this customer, and what is their customer ID, that kind of stuff, and we're passing it through our system. We're going to make a change here. We're going to actually take this very common data, and we're going to load it up right at the beginning. We're going to shove it into this passport, and have it passed down. Now, one request comes in and one request to the database, so we've actually gotten a much more linear scaling. This is important to us. Everybody along the way doesn't have to make a database request, so now it's a little bit faster. He doesn't have to request it. He's getting handed this data structure now.

You should all be thinking from the keynote, "But what if you write to it?" You want some consistency here. This is the non-linearizability that he was talking about. If mid-tier B along the way changes the plan, you're in the process of you change the plan, and now you want to render something for them. You want a consistent experience across them. If mid-tier B here - I've shown in maroon on the right-hand side here - is writing to the database and then a request goes to mid-tier C, he's still going to see the blue dots, the blue membership. That could be our inconsistent experience. If these are eventual consistent databases and we want to do better. We don't want quorum on this database, we don't want to go through all the troubles you saw in the keynote to try and give a consistent experience.

It turns out the passport has this contract where if you are changing the passport, you have to send it back up. What that means is when mid-tier B is making a change, he's going to create a new passport with this new data, send it back up to A. A is now responsible for using that data when it calls mid-tier C. Now, mid-tier C is getting a consistent experience irrelevant of the state of the database. He might get back to you next week. It might be made out of pigeons, who knows? It's going to come later. What's not shown in this slide is that the passport is actually passed up to API and up to the LSM proxy, and he can do something. He could update your cookies with this new data or some aspect of this data. It doesn't have to be the full data. We can make a reference to an immutable data ref object. We don't need the full object going around, but the key is we're passing this data up and down to this stream.

We made this small change where we're passing around some more heavy data, and we did that to put something in the layer 7 proxy, but now we get this great experience where we can get some consistent experience. This is near the land of caching, I get that. If you need to cache things, you should cache it. This is only applicable if you're going to, at the first entry point, be willing to look it up. Use caching if you need to. This is a little bit of a unique use case.

We'd got it via two trade-offs. One is data passing. We had to move past "I am an isolated mid-tier microservice," and "I will call this other microservice, and we will not share any state." I'm saying, let's switch that up, let's put a little extra data on the wire. Yes, it's nice to keep your request small, but if I grow my request size by one or two K, it's not going to take down the network. At least I haven't taken down the network yet.

The other thing here is a heavier data structure. I'm not just passing along a 64-bit int through my system, I have made a heavier data structure. All the people in the system now have to understand this passport. They need to know how to read it, some of them need to know how to change it. That's a trade-off. Now, I need to coordinate with a lot more teams, but I'm willing to make that trade-off to get this consistent database request. That's three. Let's go on to the fourth one.

Device Types

Netflix, you can watch on many different types of devices. There are over 2,000 different devices, you can watch on. We have learned that it is useful to categorize these devices, and not just individual devices, my iPad versus your iPad, but, all iPads, and all old iPads and new iPads. We want to group these things together. It looks like this. The numbers aren't important, all you need to know, there's some gobbledygook on the left, we have to convert it to some useful number on the right. That's what this service is doing for us.

Let me make it really clear why it's important to us. This is an error metric, a real one taken from yesterday. Something is going on, I don't know what device is doing it, and to my point earlier, when something goes bad, it goes bad for all types of devices. If it's for Android devices, which it turned out to be, you could see real quickly a bulk of this is Android. I could say, "Group these things together." This grouping is really important for telemetry. We need this to debug issues in production, so we're going to consider critical data that we need to have available to us. It's also considered ridiculously cacheable. This doesn't change very much. Keep that in mind.

When I joined the team, every mid-tier service would call this DTS service and say, "Do you have anything new?" When he'd reply, he'd give you 100 megabytes of XML. Somewhere in there, they felt like, "Let's put a backup system in case DTS gets falls over," which it could do because everybody is asking for 100 megabytes of XML. They put it in a PubSub system as a backup and say, "If there's a change, we'll just push it out over the PubSub system." It didn't take long to realize the backup was better than the real thing, so we just threw out the real thing, used the backup. What it did is let us take DTS out of the critical path. If DTS isn't around, no big deal. We have this PubSub system that is managed by another team, it's rock solid. It is infrastructure from our point of view, and it pushes out what was 100 megabytes of XML. We did better, we moved it to proto, and we compressed it. It got better, but the PubSub was the big thing.

Even there, we started to look a little bit deeper and realized, "What is this data? What's the real business requirement? How fast do I really need to get these updates out?" It didn't take long, you started asking them, "When does new data flow in?" "It flows in all the time, and we should see results." Ok, what is this data? This data is, someone in China wants to create a new TV, they tell Netflix about it. They want to go through certification, so in March they tell us about a new kind of TV, they start the certification process. They eventually build the TVs, they certify the Netflix app on it. It shows up at Best Buy, and then you buy it. That's March to December; that's nine-month lag before we see this device in production. This data can be nine months stale, and no one's really going to know about it.

That took a little bit of the five why's. We're, "Why do you need this data?" It turns out they didn't need it for nine months. We started to actually embed it in the jar. We had a client library, we'd ship it out, and we'd just put old data, whatever the data of the day was in there, and we had to compress it small enough at this point that it really wasn't a big overhead. Now actually, the data lives in the jar. You could still use PubSub, and you can pump out updates, put if PubSub goes down, this highly critical data is available to everybody. It might be old, it might be nine months old, but that's fine.

Along the way, we tried a whole bunch of other things like, "We'll send incremental updates." We were getting really fancy, but it was getting really complex. I'd much rather simple over easy in this case. We just ditched all those plans and just put it in the jar, and we're done.

There are two trade-offs we're talking about here. One is business tuned fallbacks. This is like really going back to the business and asking, "Exactly what do you need?" If we're going to be architects and we're looking for that freedom of moving things around, we really do need to know what they're really asking for because if they ask for something real-time - if you go back to the log all the users out example earlier, they were real-time. It didn't take long to say, "Real-time is within 10 minutes." It's a very different definition of real-time. Likewise, leveraging existing infrastructure. One was the PubSub piece, which is still important. We do like to get updates out if you are testing internally or something like that, but I'd much rather use that existing infrastructure whenever possible to have my own service where I can tune it, change it immediately. I will use existing infrastructure, that is the trade-off we made, and we were very happy with not having to get paged for a data service.

That is four use cases. Let's talk a little bit about sharding. There are different definitions of this. I'm going to give one out there, and this is runtime refactoring. If you can imagine refactoring your code, you might hopefully not change your tests. You change your code, tests stay the same. You've just moved things around. Functionally, it is the same thing. I'm talking about this at a runtime level. I'm going to take my runtime system, and I'm going to move things around, but it still has the same functionality. That's what I mean by sharding it out, so taking chunks of code and moving it around. In this context, I'm going to make reference to a message security layer. That is literally the name, you can find it on GitHub, it’s another open-source project from Netflix. It's shortened to MSL. It is a secure message sharing framework. It has encryption and authenticity and replay protection and stuff like that. It's like TLS, that's probably the best equivalent I can give you. In our case, it has some additional business things that it gives us that we want from our business, but think of it like a TLS. Unlike TLS, you can't just go to Amazon and say, "Give me a load balancer that supports MSL." You can't terminate at the load balancer, so we're going to terminate at API. Same architecture around the edge. You get a layer 7 proxy sending traffic to the first level.

This API service is doing a lot of things. It's going to terminate the security layer. It's going to run some groovy scripts because that's just how it works. It's going to have clients for everything else in the system. You saw that diagram. It spreads out really fast. This API service is going to have client libraries for everybody out there. It's a beast of a machine, it's doing lots of things, and one of the things was MSL; we had our code, and we had to get it in here, and we wanted to get it out. That's what we're going to do. We refactored it, sharded it, and took that logic out and put it into its own service. You can't entirely put it off to its own service; in our case, we have to put a small little piece in the L7 proxy. It is a security layer, but it's also an encryption. We just need to decrypt, encrypt it. We keep those simple operations up at the layer 7 proxy right there.

Of the functionality of MSL we're talking about here, there's the encrypt-decrypt, we're going to leave that at the L7 proxy, and then the more complicated pieces we got to do key exchange, we have to authenticate devices, that we're going to move off to its own service. Does all the same stuff, we've just moved things around.

We were doing this for our own operational benefits, we didn't think much would change, but then people were, "You must have messed up. Look what happened to our CPU on the way down. People must not be calling us." It turns out it just got cheaper to run API, it took less CPU to run API. We anticipated this, but we did not expect a 31% drop, "We'll take it, these are free." Right off the bat, you guys, that should to be enough. Presentation over; you should refactor your code. Then we saw it in latency, "Ok, that's interesting." It's always better to get better latency, but there isn't huge overhead to the encrypt-decrypt, or if it was, it doesn't account for this much of a change. A 30% decrease in average and a 29% in P99. If you can get a P99 reduction, that's awesome. We were really happy about that.

When you look at latency, it doesn't take long to start looking at garbage collection, and that's what we looked at, and we found this drop. The only reason you see a delay here is that we were just doing a slow rollout of the future. The top, I know there's no axes here, is a nontrivial number, like hundreds of milliseconds if not seconds down to what we consider zero. It got a lot better, the amount of time we spent in garbage collection, which is what gave us the latency, which gave us the less cost.

What we couldn't see when MSL and groovy scripts and all these things were all in one API service was, it had some really bad GC characteristics, and it was to the point where the JVM couldn't really deal with all these things going on. When we separated out, the JVM could say, "That's what you meant to do. I can deal with that, I can optimize around that." We actually are now running one MSL service and one API service where we were previously running three, because the JVM, in our case, RVM, can be more efficient now. It knows what your code is doing, it can do analysis on it. We can do analysis on it. I can now go to this MSL service, profile it, and see what it's doing. When it was over an API, too much was going on. I couldn't look at a flame graph and reason about it. By moving it out, I can now get simpler operations out of it, and that's what makes it a big win I would say - you now can reason about the operations piece of this when you have this monolith on why you should do it.

The big overhead, the trade-off is, I'm going to take a lot more operational overhead. I have to write a whole new service that is critical to running of Netflix, but my other critical service is now going to become more reliant, and that's a pretty good trade-off. We can also now separate. The person running API doesn't have to be the team that runs MSL. That was worth the trade-off in this case.

We went over five use cases that we did at Netflix, and I get that our scale isn't the biggest scale ever, but it's also not trivial. These aren't necessarily problems that you will see all the time. It won't affect everybody, I get that. I'm just hoping to give you some insight how we think about it. It goes back to that scalability, I want to make sure I give myself some leeway. If the idea is to just hit the database more in a linear fashion, I'm going to think twice about it and try and do something sublinear; something that grows at a more reasonable rate so that as our customers keep coming in and we get more memberships, we don't have to keep rearchitecting our entire system.

Bonus trade-off: do laundry. The trade-off is you have to do laundry, but the benefit here is your family loves you for it, so definitely do laundry. You can reach me at this email or on Twitter. I know I did go fast over some of this stuff, it is a lot. The lesson here is there are tools available for you.

Questions and Answers

Participant 1: For the Mantis system, you guys are aggressively assembling and doing queries and all this stuff. My question to you was, with this system, you guys are effectively dropping a lot of data that can help you form forensic analysis. How do you approach that?

Ryan: The general question is that we are filtering a lot of data, which means we're throwing away a lot of data. How do we do further analysis on this useful data? We still can. I don't have to be the only one writing a query. Anyone can write a query. There are still systems that flow off that you get recorded, but then you can change it. I mentioned that it's a query language, it has a select statement, and you can say "select this field and this field," and get very small bits of data, and the team that's interested in the small bits of data can just do 100% of all the data. I could say, select star on the small bit of data. We actually both can play in the same box and run parallel requests and still get what we're looking for. Yes, if you truly have to offload the box, maybe the agent in that diagram does still throw the data off box, but it still goes to this Mantis. That can then filter it and send it to the right places. If you can't get the benefits at the first layer of the agent, you can get it at the second layer, and that's fine.

Participant 1: What about in the eventuality that you don't necessarily know what you're looking at or looking for?

Ryan: The question is about the eventuality of what if I needed this data in the past? That's the trade-off that we're making. We have to say, "That said, we've been doing analytics for a while. We know the things that we do want." Usually, what I find is that some new service, some new feature I turned on, and it's new stuff, and that's where we're focusing most of our time, but there is some older data, historical data that's used for analytics for sure.

Participant 2: My question is the sharding part. You splitted the codes to another service, then improve the performance? I don't understand how the performance is hugely improved.

Ryan: Same amount of work is happening, so you would think this would just be literally one plus one equals two. In our case, we run in a virtual machine, and our code gets jittted over time, and so one of the things that actually happened to us was that we had memory being allocated, and in the scope of a request, the JVM couldn't tell if this data was in the eden space and was new data or it was long-lived space, because what was going on in the span of a request was so much, and when it was on two different machines running in a smaller request scope, the JVM and the jitting in the memory analysis could go in and say, "You actually don't use this memory very long. I can throw it out, garbage collection it for free, and this guy could do it too." Both of them are making much smarter garbage collection insights, and that's where a lot of the performance came from there.

It also opened up a Pandora's box. Now that we had a dedicated service, now we could start performance tuning it and getting the real kind of benefits of, tweaking your code and getting really bad hot loops and stuff like that. Some stuff for free because of garbage collection, but also a plethora of new ability to do performance tuning.

Participant 3: My question is about your passport payload. Is it something that you would put on the header? What is the mechanism behind how you would actually store that data so it would be transitive across requests? Then secondary follow-up is, when the service needed to update the passport, what was the interface mechanism to get it back? Was it in the response, or was it in some other way?

Ryan: They're two very related questions. How do we pass this passport through? A vast majority of requests are HTTP. You can send headers, so it is considered metadata as far as your request. We tried to leave the request untouched, unscathed, and it was sent as HTTP headers. In GRPC, which we also use, there is metadata fields we can use, but we also at some point said, "This is so useful. Let's just make it the request." If your service wants a passport, it wants to know who you're acting on behalf of, take that as an argument. As much as we did use it as a sideband channel, we moved towards, "Let's just move it into the request if you could." If you're writing a new service, we would ask you to put it directly in the request because if you're going to try to reproduce that request, it'd be nice to just say, "Here's the payload. Here's the passport," on the way up, it is HTTP headers on the way back also or put it into the request on the way back up.

Participant 4: I'm curious about your thoughts on performance of a serving recommendation engine-based results, specifically with two scenarios. One where they are sorted, filtered, and largely around items whose state, particularly around availability, changes very rapidly. In the scenario where an item is available for maybe two minutes, and now it goes away, or it goes into a different state, that means it needs to be way further down because it's just a different class of thing, and then it comes back a minute later.

Ryan: I don't think I follow the part about the recommendations and data changing versus becoming unavailable.

Participant 4: Specifically, recommendation around the fact that if you have all this stuff in Elasticsearch, you could be updating it and you can do your search, and that's really easy, but when it's a recommendation engine type of scenario, the sort is highly influenced by the user-item relationship.

Ryan: I'm a little bit familiar with the whole recommendation engine, it is different. It has different trade-offs for sure, and it is not very cacheable. Every customer has a different set of recommendations that we want to do and stuff. In our case, nothing I was showing is really a customer-specific. In that case, they do lean a lot more towards Memcached D-like solutions, pushing updates to Memcache D. I think it gets a little more into what the keynote was talking about and how if you want that different characteristic, you choose one of those systems.

Participant 5: For passport, you mentioned that you implemented some contract, and I can imagine that contract involves being able to deal with the same message twice, essentially message replay because you're basically taking a request that you've mutated and then shoving it back through all the previous stages.

Ryan: It does not go through previous stages, it's for subsequent requests. In the model and that pitcher at mid-tier A is going to call B, then C, then D, let's say. If something happened in B, then he called C. They're not happening in parallel, it's a serial form.

Participant 5: Right, but the preceding stages have to deal with the same request essentially happening multiple times, so if there's any side effects - how difficult was it to get to buy into the contract essentially? How much fall out was there from implementing this new contractual thing that's not really obvious?

Ryan: There are two pieces. There's the carrot and the stick. A big piece of it, if you are going to change data and you want other people to see it, you have to do this contract. If you want to be a service changing the data and it's not being reflected in other places, it's not getting saved with that cookie, if you're not getting it back up to that layer 7 proxy, your data doesn't go out, so does your service even exist? There was a reason why they'd want to participate for sure, and very few services are changing it and needing the changes reflected. We did, had to talk to very few teams on that front. The other contract was just you receive the passport and then you send it back down. This is classic just tracing, going through like that. That was very easy, we already had tracing mechanisms that we could use, and for the flowing backup part of it, it was a, if you want your data to be updated, you better send the password back up, so it melded really nicely for us.

Participant 6: My question is around the device type. You mentioned that when you have a subscription, after being used for nine months, you create a pops up model and you create a jar or something you mentioned. Do you store that pops up model and communicate back when the device come online, or in production? The entire cycle, the performance, or the management of getting into something in production from the day noticeable - how do you do that?

Ryan: Let me see if I get the question right. How do we manage getting that data pushed out to production? It flows two ways.

Participant 6: You need to store somewhere, and you need to communicate back?

Ryan: We have a source of truth, a database that can store the truth of it. There are 2,000 devices, but truly, that's not a big database. That is just a source of truth that we live, that we look at, and it's a matter of, the system makes a change to the database also knows how to push a change to us so we could push it out over the PubSub. That coordination wasn't too bad. They know they made a change, so they made sure to tell us. It does give us this dual world where the source of truth is over here, which is not production data, and we're pushing data out to production. We really want to keep those two worlds different really. The team that's optimized for making those sorts of true changes isn't the team that knows how to really run things in production, right, or doesn't have that training or that experience. I don't think I'm answering your question per se, though.

Participant 6: Do you have an orchestrator?

Ryan: Sure. They call us, I don't call them. When they make that change, and they've made a change to the UI, they tell us that there's a change. That was our decoupling. We wanted nice clean decoupling between the team that was not doing production changes to us, which is the team that was doing production changes. They would just call our service and say, "There's new data." There wasn't a lot of coordination in that front.

Participant 7: I was really impressed by the scaling from 1 million to 150 million. This is less about engineering, more about ideation - I'm sure at 1 million you guys had thoughts about how to architect this thing that you wouldn't have imagined doing at 158 million. I'm curious, from an ideation perspective, over the years your team drew inspiration about practices that you would need to learn from because obviously, your team at 1 million has a limited knowledge of what the scale needs to be and then over the years, you need to either internally be really creative or look outside for solutions. In terms of the eight years, how did you guys get inspiration for practices in ideation?

Ryan: I wasn't working on the product the whole time, but I can tell you that it comes from a few different veins. One is the willingness to rewrite things. If I'm telling you that scalability is giving yourself some wiggle room and you can grow 2X, it means that you have to be willing to rewrite it when you get to 2X and prepare for 4X. There is a willingness to rewrite. I think when people would propose, "We need to rewrite it," we were pretty open to the idea. There has to be a willingness to rewrite, there's value in that. The other half of it, I would say, is that teams would partition themselves and take ownership of it. If my team is a small team and we're seeing growing pains, that's how we know that it's time to re-architect within our smaller domain. We don't have an architect saying, "We think there's general re-architecture needed." We did it in a smaller scale, and so that really let teams really know when and how they should start thinking about the scalability issue, because they have the responsibility. If their service goes down, that's on them. They knew that pain.

See more presentations with transcripts

Recorded at:

Dec 16, 2019

Related Sponsored Content

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.


Content-Type: application/http; msgtype=response

WARC-Identified-Payload-Type: text/html

WARC-Type: response

text:
Skip to main content
padlock icon - secure page this page is secure

Problems and Prospects in Local Environmental Assessment:Lessons from the United States

Buy Article:

$54.00 + tax (Refund Policy)

ABSTRACT Local environmental assessment (EA), mandated by five US states, has introduced complex management issues for practitioners and policy makers. Based on a review of statutes, case law, and prior EA evaluations,this paper outlines three key issue areas for local EA: linking comprehensive planning and EA; making the threshold determination process more certain; and balancing mitigation with public participation. The paper argues that all the issues reflect a central tension in environmentalassessment between procedural consistency and the need for contingent responses to conditions that differ among the projects to be reviewed.
No Reference information available - sign in for access.
No Citation information available - sign in for access.
No Supplementary Data.
No Article Media
No Metrics

Document Type: Research Article

Publication date: January 1, 1998

More about this publication?
  • Access Key
  • Free content
  • Partial Free content
  • New content
  • Open access content
  • Partial Open access content
  • Subscribed content
  • Partial Subscribed content
  • Free trial content
Cookie Policy
Cookie Policy
Ingenta Connect website makes use of cookies so as to keep track of data that you have filled in. I am Happy with this Find out more

warcinfo:
isPartOf: CC-MAIN-2020-50
publisher: Common Crawl
description: Wide crawl of the web for November/December 2020
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-4.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/


text:
Labor Day weekend marks the end of summer and the start of the most wonderful time of the year for millions of Americans. No, we’re not talking about Christmas.

Christmas hasn’t been the most wonderful time of the year since Hollywood started shooting movies in color and American society, as a whole, decided to transform a religious holiday into an orgy of consumerism and a festival of awkward family gatherings.

Labor Day weekend marks the end of summer and the start of the most wonderful time of the year for millions of Americans. No, we’re not talking about Christmas.

Christmas hasn’t been the most wonderful time of the year since Hollywood started shooting movies in color and American society, as a whole, decided to transform a religious holiday into an orgy of consumerism and a festival of awkward family gatherings.

We’re talking about the start of National Football League season. Football beats baseball as the nation’s most popular sport (with a 29 percent margin, compared with baseball’s 14 percent).

Experts and pundits cite numerous reasons why football has become America’s Sport. Some say it’s the parity between the teams (27 of 32 teams have made the playoffs in the past five years). Others tut-tut that the ascendancy of football is linked to an American penchant for violence and mayhem. And still others think the popularity of football has to do with gambling.

I say the experts are all missing the real reason: Americans love football because it is the sport that emphasizes teamwork more than any other. We’re all rugged individualists who treasure our personal freedoms — but this country didn’t get built by mom-and-pop stores that refused to cooperate.

Americans invented or perfected the modern corporation, mass production, mass communication, and of course … football. We love our individualism, but when it comes to getting things done we love the sight of individuals getting together, working toward a common goal, and becoming a single team.

There is no ‘I’ in ‘team’

Every major American sport emphasizes teamwork to some extent. But baseball ultimately comes down to a duel between the pitcher and the batter — a mano-a-mano contest of strength and skill.

When Albert Pujols gets up to bat in the bottom of the ninth with the score tied, two outs, and a man in scoring position, his teammates can do nothing to influence whether he gets the game-winning hit or grounds out to third. Their contribution is limited to clapping and shouting encouragement at Prince Albert.

Basketball has always been about superstars like Jordan and Bird to some extent; last year, we were treated to a playoff team employing the sophisticated offensive strategy known as "Give the ball to Lebron and get out of the way."

Hockey … well, we’re talking about major U.S. sports here.

In football, every superstar athlete should know that his success is intimately and intricately linked to the performance of his teammates. Tom Brady, the golden boy uber-quarterback of the 2000s, set the single-season passing record in 2007.

But the NFL fan knows that without receivers to catch the ball those passes become incompletions, and without an offensive line and a pass-blocking running back, Brady ends up on his back rather than throwing for another touchdown.

The NFL fan knows that without the coaching of Bill Belichick, without the hard work of offensive coordinator, line coaches, running back coaches, film analysts, trainers and medical staff, Tom Brady never becomes the superstar he is.

I realized just how true this is when I started playing fantasy football a couple of years ago. Although fantasy football is focused purely on an individual player’s statistics, in reality you have to consider the totality of the team.

The best example is Matt Cassel, an unheralded player who had never started a college game before an injury to Tom Brady catapulted him into the spotlight as the starting quarterback for the New England Patriots for the 2008 season. He then passed for 3,693 yards and 21 touchdowns in a magical season. …CONTINUED

Judged purely on individual performance, Matt Cassel would be one of the elite quarterbacks in the NFL, fantasy-wise, in 2009. Trouble is, he was traded to Kansas City. Now his offensive line isn’t the same, his receiving corps isn’t the same, and his coaching staff isn’t the same.

Cassel still has the same talent he did in 2008 (probably), but the team around him is nowhere near the same quality. And he suffers as a result.

Why the hell do we care?

I’m sure that some of you are saying, "That’s all fascinating, Rob, but why the hell am I reading about this on Inman? Isn’t this more for ESPN or some football message board?" Bear with me.

I believe real estate more and more is becoming a team activity, not an individual game.

For example, the proliferation of agent teams over the past few years has been dizzying. While no data is available as to the precise number of agent teams in the industry today, five years ago the agent team was something of an anomaly, whereas today every brokerage company has to think about how to accommodate teams in its structure.

Anecdotally, most major markets in the U.S. are dominated by agent teams (or brokerages that act in precisely the same manner) with a division of labor between listing agents, buyer agents, administrators and support staff. At least one major company, Keller Williams, is encouraging the formation of agent teams throughout its network.

Yet, through my consulting work and elsewhere, I am finding that few teams (and brokerages) think thoroughly about what teamwork actually means. Far too many resemble a haphazard collection of individuals pursuing their own self-interest rather than a unified group working toward common goals, with incentives aligned accordingly.

An administrator is hired simply because the team leader got too busy, rather than because of a thought-out plan as to what functions that administrator will fill and how he will improve the team as a whole.

Buyer agents are brought in, but the lead-management process is ill-defined and hazy, and incentives are set up in such a way to encourage each "buyer agent" to spend as much time as possible farming his own listings.

Transaction coordinators are hired — ostensibly to help move transactions along — but often do very little except fill out paperwork, and agents remain de facto transaction managers, as well.

In football terms, this looks like a team where every play is "go long" and there’s no coordination between the offensive line, the running back, and the quarterback. It’s more backyard football and less professional football.

The difference: strategy and management

When analyzed this way, the difference between a successful NFL franchise and a mediocre one, even in the age of parity, comes down to strategy and management.

And an argument could be made that great management overcomes weak strategy every time. I submit that the difference between a successful team/brokerage and a struggling one will come down to strategy and management rather than personnel.

You could have a team of superstar agents, each one a top producer, but fail miserably, as there is no "scheme" to the business. Everyone does his or her own thing, barely talks to each other, and the organization devolves to a loose confederation of fiefdoms. The solution is strategy: a plan of attack or defense that leverages the strengths of each individual performer and minimizes the weaknesses of each. …CONTINUED

Even when a strategy is set, and everyone is playing from the same playbook, management becomes critical. It isn’t enough to set strategy; you have to manage the execution.

"Let’s capture 30 percent of the listings in these six ZIP codes" might be a fine strategic goal, but without coordinating all of the people, ensuring that each person is doing his or her job, carrying out responsibilities fully and contributing to the team as a whole, there is a breakdown in the process.

I’d like to make two observations here.

First, if teamwork — defined by strategy and management — is the key to success, then addition by subtraction is the rule rather than the exception.

Just as successful NFL franchises ship out troublesome egomaniacs despite their individual talent, successful brokerage/teams will let go of the top producer who can’t work in a team. The resulting unity of the team as a whole is worth losing that individual’s "contribution."

Second, teamwork is built on performance, not on feelings. The NFL quarterback has to be able to rely on his teammates to provide effective blocking, precise route-running and so on in order to be able to perform his job as a quarterback. If he starts questioning whether his teammates can actually perform, he will try to compensate and make mistakes.

The depth chart on an NFL team isn’t simply about dealing with injury; it’s also a reminder to each and every starter that no job is guaranteed. If the big lead generator in a team is to pass on a lead, he has to know that it will be handled by his teammates professionally, efficiently, and up to his standards of performance.

If the buyer agent hands off the transaction to a coordinator, she has to know that the coordinator will get the job done without having to look over his shoulder constantly.

The ideal team, then, is one where egos are checked at the door, where each person knows what his or her assignment is, works hard at executing that assignment, and holds each other accountable to the whole team.

It is one where strategy is clear, with clear responsibilities, clear authority and clear accountability. It is one where the leader is paying attention to managing each team member, removing troublemakers as necessary, and constantly upgrading each position to have the team as a whole be an effective unit.

Question of when, not if

I believe, based on observations of the more successful practitioners, that emphasis on teamwork is the future of the industry. This isn’t about "business models" or such theoretical frameworks. This is about getting things done, serving the customer better, and making more money.

Individual one-man shops, if successful, evolve into small teams that evolve into larger teams. Brokerages live and die by how well they can foster a team environment rather than an "every man for himself" environment.

With the growing importance of technology, of the Web, of educated customers and of access to information, it is unrealistic to believe that a single person can be expert in everything that goes into the modern real estate brokerage practice.

For teams, it’s not a question of if they will come to dominate, but rather when they will come to dominate the landscape.

You may as well start thinking about it. And successful NFL franchises are sources for inspiration and ideas. So how’s your depth chart?

Robert Hahn is managing partner of 7DS Associates, a marketing, technology and strategy consultancy focusing on the real estate industry. He is also founder of The Notorious R.O.B. blog.

What’s your opinion? Leave your comments below or send a letter to the editor.

Show Comments Hide Comments

Sign up for Inman’s Morning Headlines
What you need to know to start your day with all the latest industry developments
Thank you for subscribing to Morning Headlines.
Back to top
Up-to-the-minute news and interviews in your inbox, ticket discounts for Inman events and more
1-Step CheckoutPay with a credit card
By continuing, you agree to Inman’s Terms of Use and Privacy Policy.

You will be charged . Your subscription will automatically renew for on . For more details on our payment terms and how to cancel, click here.

Interested in a group subscription?
Finish setting up your subscription


text:
Thomas McCormack, co-owner and managing partner of Resources Real Estate, discusses how to brand a brokerage, which is extremely important in running your company or even attracting m&a offers, and the biggest challenge his market will face in 2016.

Show Comments Hide Comments

Sign up for Inman’s Morning Headlines
What you need to know to start your day with all the latest industry developments
Thank you for subscribing to Morning Headlines.
Back to top
Up-to-the-minute news and interviews in your inbox, ticket discounts for Inman events and more
1-Step CheckoutPay with a credit card
By continuing, you agree to Inman’s Terms of Use and Privacy Policy.

You will be charged . Your subscription will automatically renew for on . For more details on our payment terms and how to cancel, click here.

Interested in a group subscription?
Finish setting up your subscription


text:
Giveaway: Prayaya V3 – Portable Virtual Operating System

Prayaya V3 is a virtual operating system that you can run from a removable drive such a USB stick to launch a self-contained environment – which we commonly refer to as a ‘sandboxed’ system. Within this sandbox a user can install application software without making any changes to the host operating system. This enables Prayaya V3 to be used as a test bed for untrusted or beta software. Prayaya V3 also enables users to carry a complete desktop environment, along with their stuff such as software and files, on a USB Flash drive and use it from any computer.

Prayaya V3 can be installed on a hard disk or any removable storage devices. Since the software itself is portable, it doesn’t matter where you install it. To move Prayaya V3 from one location to another, just copy or move the entire installation folder to the new location.

If you install it on your hard disk, you will be able to install and run software, manage your files, folders, bookmarks, desktop settings on a virtualized environment independent of the host computer. If you install it on a removable device, you will be able to carry your own personal system and software with you. You can even customize the desktop and the start menu and access it from any computer.

Once Prayaya V3 is launched, a new Windows Desktop with its own wallpaper, icons, shell, etc. is started up in the virtualized Prayaya V3 environment. Any application that runs inside this environment runs off the USB device without affecting the file system of the host. On the top of the screen is a floating toolbar that allows quick switching between the host environment and the Prayaya V3 environment.

The Prayaya V3 environment is like a completely new system – you get a new start menu, a new taskbar, a new desktop and Documents folder. Any application you wish to run inside Prayaya V3 will have to installed even if it is already installed in the host system, because Prayaya V3 is isolated from the host. Once all applications have been installed, you are ready to take your system around with you.

The most significant advantage of Prayaya V3 over other portable system Ceedo and U3 is the compatiblity with applications installation. Other portable launchers require you to get the portable edition of applications you wish to run, but with Prayaya V3, you could install almost any software – even non-portable ones – such as Yahoo Messenger, Gtalk or MS Office. This is possible because Prayaya V3 comes with a mini Windows installation, maintains its own System directory and keeps its registry on the portable device. More than 500,000 applications are supported by Prayaya V3.

Prayaya V3 is compatible with Microsoft Windows 2000, XP, 2003, Vista, and Windows 7.

Prayaya V3 is a commercial software that costs $49 per license, but we have 10 licenses exclusively for Instant Fundas readers. Like always, there are no special requirements to enter this giveaway -  just leave your name and email ID in the form below. Winners will be chosen by a random draw.

The contest ends on October 12.

Meanwhile, you can download Prayaya V3 from their website. They have a 30-days trial version available.

Similar Tools: MojoPac

Be the first to comment

Leave a Reply

Your email address will not be published. Required fields are marked *


text:
phone book, or the business directory.

Area Code 684

Located in the (UTC-11) time zone. Area code 684 is the number assigned to telephones in American Samoa. The area code system is part of the North American Numbering Plan, NANP. Access to telephones in this area from telephone outside the region need to be aware of the country code, which is "1".

Information for Area Code 684

Area Code
United States
American Samoa (AS)
Major City
Pago Pago
Time Zone

List of Cities in Area Code Overlay Complex 270

  • Pago Pago

SPs and Telephone companies in Area Code 270

  • Ast Telecom
  • American Samoa Telecommunications Authority Astca
  • American Samoa License

Phone Numbers Beginning With 684


text:
intelli-CTi for Microsoft Dynamics 365

intelli-CTi for Dynamics 365 “Quick Start” Install Guide

intelli-CTi provides interactive guides, you can follow the interactive guide for the demo experience of the intelli-CTi here, or the complete experience here.

Installing the intelli-CTi Client Application

Go to install package then right-mouse click and run the client install as ‘Run as Administrator’.

Allow the installer to do the install by clicking [Next >] and [Install].

Configure intelli-CTi (Machine-level Settings)

Run “QGate Breeze Settings”, again as administrator.

Select the “QGate Integration Service for Dynamics CRM” and click [Configure…]

Select the CRM Client Integration you wish to use (e.g. Internet Explorer, Google Chrome, Outlook CRM or USD).

Then click on the ellipsis […] to create CRM Server Connection.

This is where you create the connection to the CRM system you wish to integrate with.
Click [Add]

Enter an alias/name for this connection (something relating to this connection i.e. “UAT”).

Then complete the form answering which type of deployment type (the example below is for an online instance).

Select the Region where your CRM is located.

Then click on the ellipsis […] next to Organization, this will ask for your CRM credentials and, once confirmed, it will return all of the instances available to you.

Select the organization you want to use and [Test] the connection
(you should get a message back “Connection to Dynamics CRM Server was successful”).

Clicking [Ok] will then add a connection to the listing and allow you to select this in the Microsoft Dynamics CRM Server Connection.

Select the newly-created CRM Server Connection.

For authentication, either use Default (for Windows/AD) or Specific Credentials (most common) test the connection.

Finally (once you have a connection success) click [Save] the config.

Now select the “QGate intelli-CTi Service” option and click [Configure…]

Now we are going to configure the connection to an existing intelli-CTi Service (either located within your network or cloud-hosted by QGate).

Network intelli-CTi Service
If you have an intelli-CTi Service installed within your network (e.g. to connect to an on-premises phone system), then first enter the name of the Server and click [Connect…] to test (using “Windows Authentication” in most cases).

Cloud intelli-CTi Service
If you are using the cloud-hosted intelli-CTi Service (e.g. to connect to a cloud-hosted phone system), then choose that option from the drop-down.
Then enter your intelli-CTi Subscription Key and user credentials (set up by your intelli-CTi Administrator) and click [Connect…] to test.

This is the end of the machine-level configuration (i.e. settings which will affect all users)

Configure intelli-CTi (User-level Settings)

The first time Breeze/intelli-CTi is run for each standard user, the following will happen.

When you run Breeze for the first time as the local user you will get the following message(s)
(Note: the next two prompts may appear in different orders)

Click [OK] and it will take you to the Configure telephony connections screen

Click [Add], then select “intelli-CTi Telephony Server”, it will then ask you to configure the primary extension.

You will also be asked to confirm the client integration settings for this user, as well as which CRM connection and credentials to use.

Select the desired CRM Server Connection.

For authentication, either use Default (for Windows/AD) or Specific Credentials (most common) test the connection.

You should now have a completed install.

 Related Resources:
Further Information:

See the intelli-CTi Help Center User Guidelines for important considerations of use.


WARC-Truncated: length

text:
Skip to content

Agger Outdoor Wall Sconce

by Nordlux
$ 252.00
SKU 74481031-01
Lead Time: Ships within 10 - 13 business days.
Agger Outdoor Wall Sconce features a beautiful classic design that is sure to create a stylish statement outside any home. Clean and harmonious, this outdoor wall lamp is an excellent choice for both residential and commercial exterior.
Manufacturer Model number(s) 5701581210974
Dimensions in inches and cm Height: 7.8" (20 cm) | Projection: 11.6" (29.5 cm) | Shade diameter: 10" (25.5 cm)
Finish Aluminium
Lamping type Incandescent or Fluorescent or LED
Lamping 1 x 60W E27
Material(s) Metal
Socket type E27
Connection type Hardwire
Listing Type CE Listed
Country of manufacturer Denmark


text:
New Office for Interns

As part of The ODM Group expansion strategy, we have recently acquired an office in my home city of Zhuhai. It is a very beautiful place, to my right I can see some beautiful forestry:

Interns in Asias Office View!

And straight ahead of me I can see a breathtaking view of the sea and Gongbei's skyline:

Intern in Asia coastal view!

I think you are hard pushed to find an office that has views like this!!!


text:
Boundary Breaker

Published March 22, 2015

Through his vast body of work, the late German artist Joseph Beuys blurred the boundary between artist and theorist. Opening March 13 at Mitchell-Innes & Nash in New York, a new retrospective offers an intimate glimpse into Beuys’s masterful oeuvre and influence, alluding to many of his more radical conceptual ideas such as social sculpture, in which he considered society itself as one collaborative work of art. The month-long show will present past performances as well as installations and sculptural works that date between 1965 and 1986, the year of his death. Included in this selection is one version of Beuys’s iconic 1970 piece Filzanzug (Felt Suit), a tailored suit that was modeled after his personal garments and operates as something of a ghostly self-portrait of an artist whose impact is still being felt today.


text:
ABB recommend regular preventive maintenance for inverters, IDS have formalised this requirement into a cost effective Inverter Service Contract. Here’s what’s included in the IDS Inverter Service Contract:

  • Inverter Service ContractFree Technical advice line
  • 24/7 support line with 12 hour response time guaranteed
  • Preferential access to our extensive hire drive stock
  • Maintenance report and parameters stored
  • Unused service time can be offset against other works
  • 1 or 3 year contracts available

Inverter Service Contract

Inverter Service Contracts are bespoke to each client and designed around the type and size of the inverter, the environment it is situated in and the recommend service and maintenance schedule set out by ABB.

Maintaining the inverters in accordance with the recommended maintenance schedules, ensures maximum availability, minimum repair costs, optimised performance and extended drive lifetime. This involves the use of PM (planned maintenance) kits to replace the few components that need periodic replacement e.g. cooling fans. We also operate a 24 hour a day, 7 day a week support line with free technical advice and guarantee our Inverter Service Contract holders a maximum 12 hour response time in the event of a breakdown. IDS have a large Hire Drive range, which, should the worst happen we can use to get you back up and running as quickly as possible. We will store your maintenance reports and parameters and any unused service time can be offset against other works.

An IDS Service Contract can be applied at any stage of your inverters lifetime, speak to one of our Service Engineers about an Inverter Service Contract by calling 0115 944 1036 or email

Inverter Service ContractsIDS are an ABB Authorised Value Provider (AVP). We are authorised to specify, supply, commission, install, maintain, repair and service ABB inverters.

An ABB authorised value provider deliver the authorised offering of sales, support, service and engineering in close cooperation with ABB. They bring ABB’s products and services straight to the customer’s front door.


text:
Allow us to help you and in turn, to  help others!

          Be Involved – Join the Odd Fellows and Rebekahs

Requirements for Membership

A man or woman of good character, who is loyal to his or her country and believes in a Supreme Being, the creator and preserver of the Universe, is eligible for membership. Odd Fellowship strictly forbids any interference with one’s religious beliefs or political opinions. Most jurisdictions require candidates for the adult lodges to be 18 years old.

Lodges charge an annual dues.

How to Join

Interested..... Email us and we'll get back to you.. mtlodge74@gmail.com

A candidate must be sponsored by a current member. Basically what that means is that a currect member must  sign your membership application. You don't necessary need to know a current member in order to fill out an application. If you don't know a current member, stop by  the lodge on a meeting night, call or even fill out an application online. After speaking and meeting with you, answering a few questions and answering any questions that you may have, the members will vote to accept you into the Lodge.

  • Ask people you know if they know anyone who is an Odd Fellow or a Rebekah.

  • Find the local lodge, drop in on the day of a meeting about 30 minutes before the scheduled time, and introduce yourself. We are eager to find new members and will be glad to meet you. We meet on the First, Third and Fifth Monday every month at 7 pm.

Please feel free to call anytime with any questions you may have or use the contact page on this site.

Fill out an application-

Contact Us - 717-792-3394
Voice mail - 717-685-6343

Our Lodge is located at:
525 Greenwood Rd
York, PA 17408

Lodge Benefits

The I.O.O.F. is a beneficial non profit organization committed to striving to make the world a better place in which to live.  In addition to being a beneficial lodge that supports its members, we also provide financial support to many local charities.  Dues are $31.05 semi annually, and a $40.00 initiation fee.
Some of the many benefits you will receive for belonging to Mt. Zion Lodge are:

  • Free meal the first meeting of most months at 6 PM.

  • Sick benefits in the amount of $50 for the first 13 weeks and $25 for the next 13 weeks.

  • Death Benefits for a member in the amount of $1000 per member and $500. for spouse.

  • Dues forgiveness for any member in a nursing home for extended stay.

  • College Tuition Scholarships for certain relative in the amount of $800 to $1000.

  • The opportunity to attend our many special functions at a reduced or no cost.

  • The fellowship enjoyed by all at our meetings.

  •  Free Life Membership after 45 Yrs.

  • Use of the Lodge Hall by members only for nominal donation

Some of the annual functions of the lodge are

  • Valentines Dinner

  • Strawberry Social

  • Chicken Bar BQ

  • Ladies Night Dinner

  • Christmas Party

  • Dinner Theater Trip

Use our Lodge for your club!


text:
Middleton was voted Britain's Dream Bond Girl
The British royal - formerly known as Kate Middleton - beat a bevy of famous beauties including actresses, singers and models to be crowned the public's ideal 007 sidekick in a nationwide poll by by gadget emporium MenKind.co.uk.

Paul Kraftman, MenKind spokesman, said: "We were surprised to see that Kate Middleton was the woman Britain most wanted to see getting to grips with Bond and his gadgets.

"Although her grandmother-in-law has proved that the royals are a dab-hand at super spy action, when Bond escorted her to the Olympics. It is Her Majesty's Secret Service, after all."

Kate - who is married to Prince William - narrowly beat superstar beauties Rihanna and Beyonce Knowles, while curvy socialite Kim Kardashian came in fourth place.

The fifth star Britain would most like to see alongside Daniel Craig in the follow-up to new film 'Skyfall' is Cheryl Cole, who was just ahead of sultry actress Megan Fox.

British babe Kelly Brook came in seventh place, followed by a surprise entry from outspoken singer Jessie J.
Scarlett Johansson and Kate Moss rounded out the top ten.

Britain's Dream Bond Girl top 10:

1. Kate Middleton
2. Rihanna
3. Beyonce Knowles
4. Kim Kardashian
5. Cheryl Cole
6. Megan Fox
7. Kelly Brook
8. Jessie J
9. Scarlette Johansson
10. Kate Moss

Irishcentralbox cta600 x 300px with button


text:
Suleiman Kerimov - Anzhi Makhachkala's owner, for now

They say breaking up is hard to do, and the good folk at Anzhi Makhachkala are about to find out if that's true.

For the last few years in particular football supporters, players, coaching staff and media have watched in a kind of glazed awe as mysterious billionaires have swerved into town in their million euro sports cars and opened suitcases full of money to buy up clubs, big and small, in all corners of the football playing World.

Occasionally these new-wave owners with their billions of dollars earned in often shady circumstances (There’s a reason Roman Abramovich has armed guards discreetly around him) have their motives, tactics and intentions questioned by the media. However this does not happen very often. Perhaps media members are afraid of being banned from the interview room.

Either way, the increase in individual billionaire owners, buying clubs both big and small seemingly on a whim has expanded exponentially and basically without check in the last five years.  Every so often a warning sign pops up that maybe, just maybe, having a volatile billionaire come in and buy up your football club isn’t the most wonderful thing after all.

Fans, players and staff are finding that out right now at Anzhi Makhachkala.

Anzhi Makhachkala. Pre-breakup

Reports are coming out that Anzhi owner Suleiman Kerimov has decided to make a few changes at the club, Those changes will include a drastic slashing of the club’s wage bill, major changes in staff, a shift to a youth policy in terms of players, and a seismic change in the club’s ambitions.

Effectively, after trying to break into the big time by buying up boatloads of high priced players and subsequently failing to do so, Anzhi are now going to do things the traditional way, you know, like any normal football club. They are going to try and build and earn their place in World football. Imagine that!

Sarcasm aside, this is a stunning development, and a huge, blaring, glaring, flashing neon warning sign to the football World as we watch these billionaire business men buy up clubs with little intention other than to serve their own greedy whim, a ridiculous, stubborn and foolhardy chase for glory obtained not by blood, sweat and tears but by dumping millions of dollars, euros and pounds on big players lawns and saying ‘come play at my club, I’ll make you rich’.

Changes at Anzhi are already underway.  Club president Konstantin Remchukov has confirmed that Anzhi will be undergoing a “serious shake-up” with a wildly reduced budget, slashed from the 180 million euro region down to 50-80 million, probably depending on how many players they can offload in the short term. On that note, Russian media are reporting that Russian international captain Igor Denisov has already agreed contracts to join Dinamo Moscow.

The staff is changing too. Former Manchester United staffer René Meulensteen is being replaced as coach just 16 days after he replaced Guus Hiddink. The latter must have sniffed change in the air, and got out well before this drama unfolded. Meulensteen never stood a snowball’s chance in hell in succeeding at Anzhi.

Why is all this happening? There are basically two views going around. Most of the Western media are portraying this as a salary-dump due to the fact Anzhi are off to a bad start and languishing 13th in the Russian league table, with little hope of glory this, or next, season.

However, if you dig a bit deeper, the Russian media is reporting several interesting aspects to this break up.

Suleiman Kerimov in happier times

Russian media is reporting openly that Kerimov experienced ‘stress-related health problems’ after watching Anzhi’s ugly 1-0 home defeat to Rostov on Friday and after that pulled the trigger on changes he had undoubtedly considered previously. Other Russian news entities are reporting that Kerimov suffered huge financial losses from last week’s plunge in the share price of the Uralkali Russian potash producer in which he is a main shareholder.

Isn’t it just great when a football club’s future is tied to the share price of Uralkali Russian potash?

The club itself in a blatant attempt to save face are saying that these budget changes have been made necessary by Uefa's financial fair play regulations. If in doubt, blame a French man, eh (Platini in this case)?

Where does all this carnage leave Anzhi, their players and their fans?

Right back where they started, would appear to be the answer to that question. One thing is certain. This muddle has left the transfer market with one hell of a late-flurry type shopping list of suddenly available players.

Feast your eyes, you can bet managers in the Premier League, La Liga, The Bundesliga and all over the World are doing just that.

  • Willian - 30 million euros
  • Samuel Eto 'O - 22 million euros
  • Igor Denisov - 16 million euros
  • Lacina Traore - 16 million euros
  • Alexander Kokorin - 15 million euros
  • Lassana Diarra - 14 million euros
  • Mubarak Bussufa - 14 million euros
  • Christopher Samba - 11 million euros
  • Zhusiley - 11 million euros
  • Yuri Zhirkov - 11 million euros
  • Vladimir Gabulov - 6.5 million euros

Russian media:
Official club site:
The Guardian:
NDTV sports:
BBC sports:

Follow Cormac on Twitter

Irishcentralbox cta600 x 300px with button


text:
Heart Diseases Most Common Cause of Death Worldwide, Says World Class Cardiologist, Ariyo

Heart diseases most common cause of death worldwide, says world class Cardiologist, Ariyo

By Alice Abuka, Abuja

A renown cardiologist has said that heart diseases remain the most common causes of deaths across the world.

Adeniran Abraham Ariyo, a Professor of Cardiology and Director at HeartMasters Cardiology in Dallas, Texas spoke at the launching of a memorial heart foundation held in Abuja, Nigeria’s capital.

Ariyo in his speech made available to Irohinoodua said heart disease remains the most common cause of death worldwide, and it is the most common cause of sudden death in Nigeria. Deaths from heart disease are rapidly rising in Nigeria.

Former Nigeria-s Speaker of the House of Representative, Yakubu Dogara was a Special Guest of honour.

Ariyo whose speech was read in absentia said he was delighted to be invited to the launching of the ‘TITI and KIKI Memorial Heart Foundation’, a unique foundation of its kind in our Nigeria’shistory.

He expressed appreciation to the Chairman of the occasion, Hon. Yakubu Dogara, the former speaker of the House of Representatives, the Chief guest of honor, Honorable Senator Eyinnaya Abaribe, the Senate Minority Leader, and the Special guest of honor, Honorable, Pauline Tallen, Hon. Minister of Women affairs.

Ariyo said “I also want to give a special thanks to the convener of this event Hon. Yusuf Tajudeen, the founder and executive secretary of this foundation.
I would have loved to have been present in person at this wonderful event, but because of the pandemic, I could not be with you in person, but I am with you in spirit.”

He said he was privileged to take care of the late Mrs. Titilayo Yusuf during her period of stay in Dallas, and at Baylor Scott & White Medical Center until she went to join the Lord.

He said he was excited to join in the worthy cause at this event, as this would make her proud with a big smile from heaven beaming down upon us.

He said “I am a Nigerian and I attended the University of Ibadan before moving to the United States 30 years ago. During this period, I have dedicated my life to the care of patients, especially those with heart diseases. I am therefore very excited that this TITI and KIKI foundation, dedicated solely to the care of heart patients is being launched and it warms my heart and I’m delighted.”

He said further “My colleagues and I who are trained in the prevention, the diagnosis and treatment of heart disease consider reducing cardiovascular diseases our goals too. Thus, our goal is intertwined with the goal of this heart foundation. Therefore my colleagues and I are looking forward to a working partnership with you to ensure you achieve your goal. I thank you for your foresight, your determination to make Nigeria work despite the difficult terrain, and your perseverance and your optimism.”

He said the vision of the Foundation which is to eradicate heart disease and your goal of reducing the burden of heart disease is commendable.

Please enter your comment!
Please enter your name here


text:
Isabel Villanueva creates Pamplona Reclassics Music Festival

May 2020 Isabel Villanueva has announced a new project in her versatile career : Pamplona Reclassics International Music Festival. Founder director of a new summer music festival in her hometown, Pamplona (Navarre, Spain), with inspiration of ‘Re-discovering Classical Music’ and taking ‘Classical Music into a new step’, the spanish viola […]

Viola Power International Course 2020

Welcome to the first edition of the Viola Power International Course, to be held in Granada from 3 – 6 September 2020. (New dates after having postponed the course due to the situation with Covid19). The course welcomes international viola students (over 16 years old) in a professional, bachelor or […]

Isabel Villanueva debut in Latin America

This season 19/20 will see Isabel debut in Latin America! In september she will be performing recitals at historical Sociedad Filarmónica de Lima and Club Nacional de Lima (PE), presenting her álbum Bohèmes together with François Dumont. She will be guest artist at Interviolas Festival in Bogotá (CO), holding masterclasses […]

Isabel Villanueva appointed Viola Professor at ESMAR in Valencia

Isabel Villanueva has been appointed Viola Professor at Escuela de Música de Alto Rendimiento (ESMAR) in Valencia, starting in September 2019. The new official private music institution, is formed by highly qualified team of professors and will train students for Bachelor, Master and other complementary degrees. Application starts from April, […]

Isabel Villanueva guest artist at Prix des Lumières in Paris

Isabel Villanueva was invited to perform at the ceremony “Prix des Lumières de la presse internationale” on 4th February hosted at Institut du Monde Arabe in Paris. She performed viola solo music by Manuel de Falla and a tribute to Claude Lelouch who was in the gala, with the cover […]

Isabel Villanueva launches #ViolaPower

Isabel Villanueva continues sharing her passion to promoting the Viola, she has just created the #ViolaPower registered brand. The concept design has been made by renowned artist Mikel Urmeneta, considered one of the “100 top world creative artists” by Forbes. The first step of this project starts launching a brand […]

Isabel Villanueva receives ‘Navarra Television’ Young Value Award 2017

On 30th November a ceremony was celebrated at Baluarte Auditorium in Pamplona hosting the II edition of Navarra Television Awards, which recognized Isabel Villanueva as one of the most important violists of her generation. Born in Pamplona in 1988, Villanueva is specially grateful for having been awarded this prize which […]

‘Bohèmes’ album release : 20th October 2017

Isabel Villanueva will present her debut album ‘Bohèmes’ on 20th October in a press-conference held at Teatro Real in Madrid. This is her first recording project for viola and piano recorded with François Dumont for IMMStage label. The album includes works by Liszt, Granados, Hahn, Martinu, Enesco and Debussy, and […]


text:
One-off Storytelling Works ...

Obviously, we’re passionate about applying storytelling techniques to our client campaigns.

These are more than words.

We walk the talk.

Here’s an example from Jacqueline, one of our senior account executives in our Silicon Valley office.

By Jacqueline Velasco

The Hoffman Agency (Silicon Valley)

Being solely reliant on press releases for coverage is a thing of the past.

As I have learned first-hand, the best client placements come from one-off storytelling.

Really looking at what your client has to offer – whether directly about what it is trying to sell or a more unique story on internal items – can make a difference in the quality of coverage you secure.

So, where can you find these types of opportunities to tell your client’s stories?

My favorite would have to be Twitter.

Following those that care about the client’s industry or those from mainstream media – and monitoring them daily – is key. This way, you can find relevant opportunities that otherwise would be missed.

Along this line, we recently landed our client, Coupa, in an Inc. story titled – What I Pack: Inside the Bags of Road Warriors. I particularly like this example because it offered the opportunity to show the company’s humanity.

As far as how the piece came about –

I’d followed and worked with John Brandon, a contributor at Inc., and had tweeted him on other story angles in the past for his slideshow stories.

storytelling tweet 1

storytelling tweet 2

By developing a relationship with him via Twitter prior to this opportunity and establishing myself as a good source for his pieces, I was able to get the heads-up in advance that he was looking for interesting things executives take on their travels.

Sometimes clients can make landing coverage like this difficult because it doesn’t directly tie to selling their products. Fortunately, Coupa views PR as a tool for building the company’s brand; consequently, it appreciates the value of cracking the mainstream media even with stories that stray from its industry of spend management.

We packaged up the opportunity to Coupa by asking for a photo depicting all the items the CEO Rob Bernshteyn MUST have when he travels as well as why. Letting the client know that anything goes – if you travel with it include it – no matter what the item may be, and the more creative the better. Since the final product was a slideshow, the photo was a necessity for inclusion.

Within a few hours we had the following photo of all the items Rob takes on his travels sent over to John:

Coupa travel items

Knowing that John would be inundated for a story like this, we knew making the final cut depended on being creative with the photo and commentary:

So, what is in his bag? Let’s take a look:

  • MacBook Pro computer: Rob takes advantage of his flight time by using his computer to catch up on work while on his long flights.
  • T-Mobile Droid: To stay connected with his colleagues and family while he is away, Rob makes sure he has his mobile at all times. He even takes advantage of the apps on his phone to keep himself entertained.
  • Plantronics headset: Rob is quite the multitasker! He relies on his Plantronics headset to keep his hands free during calls so he can take notes.
  • Business cards: Rob makes sure to carry business cards with him on all his travels. You never know when you will run into a potential customer on a flight.
  • Isotronic adaptor: His adaptor is a must have for his international travel to Europe since it allows him to power up when on the go.
  • Seaweed: Rob does not go anywhere without his seaweed. For him it’s an excellent source of vitamins and minerals that balances the Red Bull.
  • Red Bull: No explanation needed. It’s what Rob refers to as a necessity for long trips.

Taking a picture and teasing out the storytelling behind each item was key to securing this coverage.

And the value of this coverage extends beyond the individual placement. John Brandon knows he can count on us to deliver the content he needs for his articles, while other influencers see that the Coupa CEO is willing to show more than a boilerplate view into who he is as a person.

Press releases and product news simply cannot deliver this kind of exposure.

Plus, one-off storytelling offers more room for creativity, and speaking from my own experience, is simply more fun.

Leave a Reply