This browser is not actively supported anymore. For the best passle experience, we strongly recommend you upgrade your browser.

NEWS & INSIGHTS

| 13 minute read

Building Cyber Engineering Teams Overseas

When Boris Chen and I first started our company, tCell, we split our engineering team between Stockholm and San Francisco. At the time this was an extremely unusual move, but it worked well for us. Boris had a long history of building and managing teams in Stockholm and the talent pool there was uniquely suited to the specialized technologies that our product was built on. It was not without its challenges, though - legal, cultural, process and management were all harder. But we as climbed the learning curve, we found the tradeoff for access to talent worth the added friction.

Today, building and managing remote teams has become commonplace for startups. Zoom, Slack, COVID and the explosion in Bay Area FANG hiring and compensation have combined to make it both easier and more compelling for early-stage startups to make overseas development a part of their early company building strategy. But as advantageous as this strategy can appear, the mechanics of executing it are often daunting to entrepreneurs. For founders of cyber security companies, the challenges are amplified by the specialized knowledge needed to build our products and the scarcity of that expertise globally.

At AllegisCyber, roughly half of the companies that we’ve partnered with recently have built overseas development teams and we expect that portion to grow. Those that have gone through the process have accumulated valuable experience in how to make it work, maximizing upsides and minimizing risk. I’ve tried to capture some of their key lessons here.

First: Try to talk yourself out of it

When building a technology startup, you win or lose with the quality of your product and the market’s need for it. When you get that right, everything else – sales, marketing, finance, etc., becomes a matter of execution. It may be hard, but it definitely can be done. On the other hand, if your product sucks or if it isn’t needed, no amount of cost savings or go to market brilliance will save you.

To build a phenomenal product, you need phenomenal engineers and a close connection to customers. The easiest way to do this is to hire the best engineers in the world and put them in the same room with the people talking to your first customers. Communication is automatic and execution is natural. So, the first bit of advice I have is that if you can build your team where you are (Bay Area, Tel Aviv, wherever), do so. There’s a cost to going remote and a bigger cost to going overseas. Make sure it’s worth it. And more specifically, if you’re going to trade off on agility by going overseas, make certain that you are trading up in quality for the roles that you fill. 

With that in mind, the considerations I’d suggest when thinking about whether to build an overseas engineering team are all centered around whether you meet your hiring plan for engineering where you are (Bay Area or otherwise):

  • Can you compete for talent?  Maybe you need 15 engineers, a few with specialized knowledge of your area of infosec and the rest full stack engineers with experience building reliable, scalable cloud analytics platforms. Setting salaries aside, those are skills that are hard to find, in high demand, or both. If you’re in the Bay Area, for example, will you be able to poach experienced engineers from a FANG company or an AI startup that just landed a $200M funding round? For our company, we could get those people but not enough of them fast enough.
  • Are people with the needed skills somewhere else?  Full stack engineers, devop people, data pipeline experts are all skills that are likely common in your city. But sometimes you need very specific skills. If you’re building a RASP like we were, you want people familiar with language and VM internals. If you’re building an OT security platform, you’ll need a core group of people familiar with manufacturing technologies and the security issues surrounding them. Sometimes those people congregate in other geographies for historical reasons – in Stockholm, where BEA happen to have a JVM team, or in Maryland where lots of people were trained by the NSA. If you need these skills and the best people in the world are somewhere else, that pretty much answers the question of whether you should have remote teams.
  • Costs – This is obvious, but also the trickiest. As a rule, seed funded startups cannot pay $200-500k total comp that an experienced engineer might get at a FANG or similar company. They just don’t have the money. On the other hand, a trap founders fall into is to think of overseas development as a pure cost savings measure. A common refrain I hear is “I can get three people in over there for the cost of one engineer here!” That may very well be true, but three crappy engineers won’t advance your product faster than one great one. For an early stage company building technical security products, the most important thing is to build the highest quality, highest performance team you can, given immovable constraints. It’s fair to start a remote team to build the best team you can afford, but it is a mistake to start a remote team to build the most “cost effective” team you can.

How to build the team

Getting started with an overseas engineering team can be daunting. There are a number of critical decisions to make that are both high impact and difficult to reverse. Among them

  • What work to assign to the team
  • How and where to build the team
  • How to setup the team legally

These questions are all interrelated and there is no one answer to any of them. However, there are a set of defaults you can use, deviating when there are specific reasons to.

What to assign the team.

Sandeep Lahane of Deepfence recommends splitting your work between generic engineering tasks and those requiring specialized infosec background. The majority of your work will require more generic skills that are available lots of places. This may include UI, back end data analytics, devops and similar functions. They’re hard to scale in the Bay Area and are readily available in many other geographies. A common approach is to build generalist engineering teams overseas and keep the specialized infosec skills close to home.

It's also important to make sure the overseas teams can function largely independently. The worst killer of productivity (and joy) is to work on something urgent and get blocked on someone who’s asleep and won’t respond to your messages for hours. Overseas teams should be given all that they need to ship capabilities without blocking on input or work from people in another time zone.

Team leadership

Hiring and managing overseas team is hard. Cultural and practical differences can create landmines that are difficult to avoid without deep local knowledge. For that reason, it’s generally smart to start off finding a local team leader for your remote team before starting on a hiring spree. That team leader needs to fully understand the goals and challenges of your company, both technical and business, and to become the primary conduit between teams. Unless one of your co-founders can fill this role, this hire will be one of your company’s most critical.

Your ability to find a leader that you trust also may drive the location of your team. If your co-founder knows Bangalore, do Bangalore. But if you’re going to hire a new team leader, consider looking in a number of acceptable locations and build your team based on the candidates you find.

Getting this hire right is make or break. We have at least one portfolio company that has a largely failed overseas team strategy because the leader they hired to build the team lacked the right technical and leadership skills needed.

Recruiting

Recruiting overseas is initially extremely hard. Unless you have a team ready to start, plan on it taking 2-3 quarters to get up and running. To give a sense of what’s involved if you set your bar appropriately high, Pankaj Rajan of MarkovML interviewed 100 people over five months to hire their first four engineers in Bangalore. They also put great effort into differentiating themselves from other startups – refining their story and culture while weeding out the mercenaries in their pipeline with a lengthily and intensive interview process. They’ve grown several fold over from there and now have a world class engineering team there, but it was far from quick or easy.

Outsourcing

I have a knee-jerk negative reaction to early cyber startups outsourcing development. At the beginning, product is the company and the technical team is the current and future product. Outsourcing that just seems wrong. However, my thinking has been challenged in two specific scenarios. Early at Okta, we outsourced connector development to a shop in Costa Rica, and development of a small mobile app to a shop in Ukraine. Both were successful by quickly addressing dev needs with good quality and reasonable management overhead. The key to both successes were that the problems were well-defined and tightly scoped. We could hand a clear, highly specific spec to the team and expect finished results in days to weeks with little iteration. Equally important, the skills they built were not core to our IP. The IP was in the design and process (defined at HQ), not the code. That meant we weren’t outsourcing what really mattered to the company.

Piotr Kupisiewicz at Elisity showed me a great example of another approach to using outsourcing for overseas development. Piotr’s strategy to quickly build a team in Poland focused on UI and data analytics was to directly hire a core set of engineering leads directly, but to use a software development house to provide long term dedicated heads to augment the team. This had a number of advantages over attempting to hire his full team directly. The service provider worked more like a recruiting firm than outsourcer in that Piotr and his leads treat the contractors as individual contributors working as full-fledged members of the team. Engineers are assigned to work for Elisity long term after passing an interview process largely the same as that used for direct hires. Additionally, because the contractors are employed by the service provider, it’s far less difficult to correct a bad hire than if the person had been hired directly. The service provider is happy to swap one person for another and the person who leaves just gets assigned to another client. Ideally, your agreement with the provider will also allow you to convert contractors to direct hires after a defined period (for a usurious fee). Piotr’s approach to at Elisity showed me that you can have an outsourcing strategy that gives equivalent quality and ownership with faster recruiting. Cost may be a little higher, but Piotr probably save a full quarter in ramp time and gained in ongoing flexibility.

Where to build the team: Where you build your team should depend primarily on the leader you choose, but other considerations are important. Pick a geography with a critical mass of available talent (general or specialized). Small markets may look appealing but will cause problems as you scale. Cost is also a factor. New York is great for talent, but probably not the best place to go as your second geography after San Jose. Legal issues are also a potential factor, particularly related to employment law. You’ll want to ensure that you have good agility in hiring and firing and that you can issue stock options to employees without causing tax issues for the company or the employee. Talk to a local attorney with relevant experience early in the process to avoid difficult to reverse mistakes.

Here are notes on common options:

  • India: Bangalore, Pune, Chenai, Hyderabad are all possible. Bangalore has, by reputation, the best startup culture but it is also the most competitive. Compensation is about ½ that of Bay Area and growing. Tax and legal issues are minimal. Get someone local to set up your entity and issue US stock options.
  • Eastern Europe: Poland is the center. Generally great work culture, lots of good talent with fewer opportunities available. US and Israeli startup jobs are highly sought after. Comp is 1/3-1/2 that of Bay Area. Employment law is more restrictive, particularly when firing someone, and taxes can be quite high. Your first 10 or so engineers will want to form consulting businesses that you’ll contract with rather than have you hire them directly. This may not scale, but is hugely advantageous to you and the employee. You’ll need local council to draft the contracts and decide on how to issue stock to avoid messy tax issues. Use someone who’s done it many times before.
  • Western Europe: Great talent in most cities. We had success in Stockholm, but other cities can work. However, costs are closer to the Bay Area (2/3), so probably only worth pursuing if you already know the people or if there are specialized skills only available there (e.g. ARM internals). Do the same contractor structure as Eastern Europe, but be extra careful with stock as Sweden and other countries have even more complicated laws.
  • Latin America: At Okta, working with our Costa Rica outsourcing team was very convenient because of the time zone, but skills were limited and we did not increase that investment. I’ve heard rumors of more success in Mexico and Brazil, but I have no specific examples of this working.
  • Smaller US/Canada markets: Maybe, but if you’re doing this it may make more sense to just do distributed teams and not think of this as a separate engineering location. 
  • Do no not touch: If you’re a cyber security company, keep your teams in countries closely aligned politically with the West. This obviously means no to Russia and China. But it also means a hard no to countries whose alignment is less certain.

What about going fully distributed?

That’s a different dimension that I have less experience with. A fully distributed US team certainly promises to address the talent pool issues associated with sticking with one major metro. Costs savings are not likely to be material for the first 20 engineers, but that should not be the deciding factor anyway. The big issue is culture and productivity. I don’t know that we as an industry have cracked the nut on how to build creative, high velocity, high quality teams fully remotely. Perhaps that’s for another post to explore. In the context of this post, though, I would not recommend a fully distributed global team. Time zones would force fully asynchronous work and cause everything to go much more slowly. As you scale, it will get worse, not better. I don’t think that the trade-offs make sense.

Tricks to making overseas teams highly successful

  • Be deliberate about building a network. Your star performers probably know other star performers from previous jobs or school. Referrals are even more important for overseas sites than locally.
  • Get to know the local universities. Start an intern program early.
  • Travel frequently for in person meetings. An HQ founder should visit at least twice a year if not quarterly. Your team lead should be in HQ at least quarterly. Budget for this.
  • Rotate engineers to HQ for multi-month stays, ideally in groups. Rent a hacker house (Boris rented a houseboat in Stockholm once). This is super fun and great for establishing deeper working relationships. Also rotate people from HQ to your overseas site.
  • Be generous with equity. Market rate for equity grants overseas is often a lot lower than the US. But you can create enormous loyalty and brand awareness for hiring by being more aggressive here. Your company’s success should be life altering for all of your engineers, wherever they are.

What not to do/what to watch for

  • Two locations max. Coordinating across three is way too hard for a small company. We have seen portfolio companies try India, Bay Area and UK, for example, and it failed because communication broke down.
  • Don’t split QA and Dev. This requires far too much coordination.
  • Keep the work as self contained as possible. Don’t make your overseas team reliant on your local team to deploy code. Again, too much friction.
  • Watch output carefully, both from individuals and the team as a whole. If you see signs that things are not working, they’re probably worse than you think. Fix it immediately.
  • Don’t compromise on English proficiency for any employee. Communication is hard enough with time zones and culture to negotiate. Adding language gaps will make it impossible to work at a high level.

Will recent layoffs change the equation?

Short answer: Maybe, but probably not enough.

Big tech companies have laid off 100k people in the last few quarters. Most startups have reduced staff or at least slowed hiring. VC funding in cyber is down 60% from its peak. In core markets like the Bay Area, Tel Aviv and New York, this should naturally result in more available talent and lower compensation. I have not seen specific data to show that it’s happening, but it doesn’t seem plausible that the effect won’t be there.

That said, there are a number of reasons to think that the basic logic of building an overseas dev team will continue to hold. Firstly, the talent crunch in core cities has been building for years. We decided to build our Stockholm team at tCell in 2015. Layoffs may wind the clock back a year or so, but we’re never returning to pre-2015 conditions. Secondly, we’ve crossed the chasm in terms of how to build and run overseas teams at small scale. The talent pool has truly become global and the communication tools are highly refined. The trade-offs just aren’t as bad as they were pre-COVID. Lastly, building a team overseas is a long-term strategic bet. A US downturn now may make it possible to hire a handful more engineers locally and delay starting the overseas process, but it won’t change the overall trajectory.

Need help?

If you’re exploring starting an overseas team and are stuck on anything, feel free to reach out. I may not have the answer myself, but there’s a good chance I know someone who’s solved a similar issue and is willing to share. m@allegiscyber.com / @michaelf@infosec.exchange.

Tags

blog