Perspectives · Data Operations
No Data Engineering Team? What Mid-Market Hedge Funds Do
Sub-$2B hedge funds rarely have a data engineer. Why fragmented data persists, which makeshift fixes work, and how ops can run pipelines without a hire.
· 13 min read
If you run technology or operations at a hedge fund managing under $2 billion, you almost certainly have engineers on staff. What you almost certainly do not have is a data engineer. More precisely, you rarely have someone whose entire job is building and maintaining the data pipelines that move position files, NAVs, and counterparty feeds from where they land into where they are useful.
Most sub-$2B funds employ capable technologists who handle trading systems, execution, infrastructure, and the general plumbing that keeps the firm running. But data engineering, as a dedicated discipline, tends to be folded into that broader workload. One engineer might own the execution platform and also get pulled into reconciling a prime broker feed at month-end. Another might be a quant developer who inherited the data pipelines because no one else had the bandwidth. The result is a team that can do everything, but where data work competes with everything else for attention.
This is the mid-market reality, and it is the root cause behind a pattern most practitioners will recognize: fragmented data slows firms down. The firm has the talent to solve the problem in theory. It does not have the specialized capacity to solve it in practice. This article is about what that gap looks like day to day, why it is getting harder to close with generalists alone, and what firms can do about it.
What "no dedicated data engineering capacity" actually looks like
The phrase "no dedicated data engineering capacity" does not mean a firm has no engineers. It means data engineering is not a standing function with an owner, a roadmap, and the time to do it properly. At a mid-market hedge fund, that shows up in recognizable ways.
Your team is reconciling multiple prime brokers in spreadsheets, chasing breaks over email, and rebuilding reports that should already exist. Nearly 70 percent of asset managers cite manual effort as the top bottleneck in their end-to-end processes1. Month-end reporting means a week of data pulls, cross-referencing, and formatting before anything goes to the CFO or investors. Every new fund launch or strategy means rebuilding the spreadsheet from scratch, or worse, inheriting a model no one fully understands.
The reliance on spreadsheets is not a fringe practice. According to a 2025 transformation survey by the consulting firm Citisoft, 52 percent of firms still use spreadsheets for regulatory reporting, and one in three of those rely solely on spreadsheets, with no supporting systems or external partners2. The same survey found that firms with under $100 billion in AUM face structural blockers including fragmented systems, spreadsheet reliance, and limited internal expertise that slow execution2.
The consequences of poor reconciliation are well documented. In one striking example, Archeus Capital Management filed a $465 million lawsuit against its administrator, GlobeOp Financial Services, accusing it of "colossal failures" in trade reconciliation and neglecting to provide timely statements to investors. Archeus saw its holdings shrink from a high of $3 billion in May 2005 to $700 million before closing in October 2006, and the lawsuit alleged that the reconciliation failures were largely to blame for the investor redemptions that led to the firm's closure34. While that case involved an external administrator rather than an internal data engineering gap, it underscores what is at stake when reconciliation breaks down: investor confidence, and ultimately the firm itself.
Why this is a mid-market problem, not just a small-fund problem
It would be easy to frame this as a challenge only the smallest funds face. The evidence says otherwise. The talent market has shifted in ways that make dedicated data engineering harder to staff at every level, and the mid-market feels it most acutely.
Hedge funds rushed headfirst into the AI arms race, but now they face an unexpected hiring challenge: too many AI specialists, and not enough candidates for core engineering roles. As top quant and multi-strategy funds built out AI infrastructure, the market became saturated with machine-learning-focused candidates, while interest in core engineering roles declined sharply, leaving a gap in key non-AI positions that underpin trading systems, data pipelines, and infrastructure5. According to recruitment firm Affinity North, it is getting harder to find good junior and mid-level software engineers to work on non-AI related projects, as engineering folks in Big Tech who would have moved over to trading have pivoted to AI6. Demand for specialized skills, including data engineering, has surpassed available supply, and firms are offering multi-year guarantees, relocation packages, and "team lifts" to win candidates7.
Mid-market funds cannot compete on compensation the way large platforms can. The people who can build robust, governed data pipelines are drawn to firms that can pay top of market and offer them a dedicated team to lead. A sub-$2B fund, running lean by design, is not winning that bidding war.
The practical consequence is that mid-market funds rely on hybrid roles, and those roles get overloaded. At Kvartil, a Stockholm-based boutique asset manager, Chief Operations Officer Jacob Werkell described the challenge as "maintaining a clean, consistent, and timely data foundation across the entire value chain, from trading and risk to client and regulatory reporting, while the firm grows in products and jurisdictions without increasing headcount at the same pace." He acknowledged that "if we are not careful, that can lead to fragmented data, manual checks, and spreadsheet workarounds"8. That’s the mid-market data engineering gap: the firm is growing, headcount is not, and data is fragmenting.
The cost of the workaround
When data engineering is absorbed into generalist work, the cost shows up in time, risk, and opportunity. Reconciliation is one of the clearest examples of how manual processing turns into margin leakage. On paper, reconciliation is a control function: ensuring that positions, cash balances, transactions, fees, and reference data align across systems and external parties. In practice, when reconciliation is manual, it becomes a recurring tax on the business9.
The direct labor cost is significant, but the opportunity cost is often greater. Skilled operations team members become consumed by repetitive tasks, reporting cycles slow down, errors become more likely, and audit and compliance reviews become more burdensome because evidence is scattered across systems and spreadsheets9. The margin pressure is real. PwC's 2025 Asset & Wealth Management Revolution research reveals that profit as a share of AUM has fallen roughly 19 percent since 2018, even as global AUM keeps climbing9.
A pricing discrepancy, stale reference data, incorrect fee calculation, or delayed reconciliation can affect portfolio reporting, client communications, regulatory filings, and internal decision-making9. According to research by the Alternative Investment Management Association, 51 percent of firms struggle with data quality, particularly when aligning with unstructured documents, given that many term sheets, confirmations, and regulatory filings still arrive as PDFs10.
The broader industry context makes the timing pressure worse. The United States sped up its settlement cycle in May 2024, requiring that securities transactions settle one business day after the trade, instead of two. A Citigroup survey found the shift had a greater-than-anticipated impact on market participants across the board11, and the DTCC advised hedge funds to consider adding automated post-trade solutions because the time to automate was now12. Reconciliation is no longer a back-office hygiene exercise; it is a front-office risk function, and the window to resolve breaks has shrunk dramatically. A generalist engineer who is also supporting the trading system does not have the bandwidth to own that process to the standard the new timeline demands.
There is also an allocator-facing dimension. When an allocator asks how a firm ensures data integrity, the answer needs to be more than "we have a smart person who handles it." Full data lineage, the ability to trace a number back to its source, is what gives a firm a clear, confident answer during due diligence. Without a dedicated owner for data, that lineage tends to live in people's heads rather than in a system, and people leave.
Makeshift approaches firms try without a vendor
Many firms rightly start trying to close the gap with what they already have. These makeshift approaches can buy real time. They are not, however, a substitute for a dedicated data engineering function, and it is worth being honest about where they help and where they stop.
Standardize file templates and naming conventions
The single highest-leverage thing a mid-market fund can do without spending money is impose order on inbound data. Agree with each prime broker and administrator on a consistent file format, naming convention, and delivery schedule. Much of the pain in reconciliation comes not from the data itself but from the fact that every counterparty delivers it differently. A few hours spent negotiating templates can eliminate a class of breaks before they happen.
Build a lightweight reconciliation checklist
Document the reconciliation process end to end so it is not dependent on one person's memory. A checklist does not automate anything, but it makes the process repeatable and transferable, reducing key-person risk and making it easier to onboard new operations staff.
Designate a "data owner," even informally
One of the reasons data work drifts is that no one owns it. Naming a data owner, even if that person already has other responsibilities, creates a single point of accountability. This does not require a new hire. It requires a decision: this person is responsible for the integrity of the data, and when something breaks, they are the first call. The risk is that this person becomes the overloaded hybrid role described earlier, so the designation needs realistic expectations.
Use scheduled scripts and macros to reduce manual steps
Most mid-market funds have at least one engineer who can write a script. Automating the most repetitive parts of data collection, file movement, and basic validation can take hours of manual work off the operations team. This is where generalist engineering capacity can genuinely help. The limitation is that scripts tend to be brittle. They break when a counterparty changes a file format, and the person who wrote them is often too busy to fix them quickly.
Document lineage in a shared wiki
Even a simple wiki page that traces where each data set comes from, how it is transformed, and where it lands is better than nothing. It gives the firm a starting point for understanding its own data and makes due diligence questions easier to answer. The catch is that a wiki is only as good as its last update, and maintaining it is precisely the kind of unglamorous work that falls off the list when the generalist engineer is fighting fires.
These approaches are cheap and sensible, but they depend on people who already have full-time jobs. They reduce the friction of fragmented data, but they do not eliminate it. They are stopgaps, and stopgaps have a shelf life.
When the stopgaps stop working
Growth is often the catalyst and barometer for change. A new strategy means new counterparties and new data sources. A new fund vehicle means a parallel set of reconciliation processes. An allocator's due diligence questionnaire gets more detailed as the firm's AUM rises. T+1 has already compressed the timeline, and any further regulatory tightening will compress it more.
At a certain point, the makeshift approaches break under the weight of volume and complexity. The checklist that worked for two prime brokers does not scale to four. The script that handled one administrator's file format breaks when a second delivers something subtly different. The data owner who was managing fine at $500 million AUM is underwater at $1.5 billion.
The AIMA research illustrates what happens when fragmentation meets market volatility. It describes a European hedge fund during the August 2024 yen carry trade unwind whose AI analytics should have provided early warnings. Instead, its fragmented trading and risk systems delivered conflicting signals, while legacy workflows delayed critical data updates. The result was preventable losses10. When data is fragmented and no one owns the pipeline, the firm does not just lose efficiency. It loses the ability to see what is happening in time to act.
This is the gap between "we have engineers" and "we have data engineering." The firm has not failed to invest in technology. It has invested in generalist capacity, which is the right call for a lean organization. But the data problem has grown beyond what generalist capacity, applied part-time, can solve. The question is how to address it without hiring a dedicated data engineering team that the firm's economics may not support.
The aspirational frame for what comes next is straightforward: all your investment data, accessible, actionable, and AI-ready. That is the state mid-market firms are reaching for, because it is the state in which the data work stops being a tax and starts being a foundation. The challenge is getting there without adding a team of data engineers.
This is the gap between "we have engineers" and "we have data engineering."
How Aquata changes the equation
This is where Aquata, Arcesium's investment data platform, helps firms solve this problem. The premise is not that a firm should hire more engineers. It is that the people already in the seat, the operations and business teams who understand the data, should be able to manage data workflows themselves, without a dedicated engineering team.
Aquata is built around no-code pipeline transformations, a form-based Data Quality rule builder, and self-service reporting tools, including Data Explorer and Query Studio. These are designed so that business and operations teams can build and maintain data workflows, define data quality rules, and produce reports without writing code or filing tickets with an engineering team. For smaller funds whose generalist engineers are stretched thin, this directly addresses the bottleneck: the data work moves off the engineering queue and into the hands of the people closest to it.
The platform consolidates data from every prime broker, administrator, and counterparty into a single source of truth, so breaks surface automatically rather than being hunted down by hand. Full data lineage gives a firm a clear, confident answer when an allocator asks how it ensures data integrity. And because Aquata is purpose-built for investment data, it fits into existing workflows rather than requiring a firm to rebuild its stack.
The point for a mid-market firm is not that Aquata replaces engineers. It is that Aquata alleviates the need for a dedicated data engineering function by giving the existing team the tools to do the work themselves. The generalist engineers stay focused on trading systems and infrastructure where they add the most value. The operations team gets a way to manage data workflows without waiting on engineering. And the firm gets to transform fragmented data into action without the overhead of building a larger team it cannot justify.
For a fund that has outgrown its stopgaps, this is a natural next step. It makes data usable and lets the people who understand it act on it.
Sources
- 1.Beating the Breaks: Modernizing Managers' Reconciliation Models, Alpha FMC awm.alphafmc.com.
- 2.2025 Transformation Survey Report, Citisoft citisoft.com.
- 3.Globe Stop? Lawsuit Puts Hedge Fund Administrators in the Spotlight, Institutional Investor institutionalinvestor.com.
- 4.GlobeOp Settles $465 Million Lawsuit Brought By Archeus Capital Management, Global Custodian globalcustodian.com.
- 5.Hedge funds face AI hiring hangover as core tech role talent pipeline dries up, Hedgeweek hedgeweek.com.
- 6.Hedge funds' new hiring problem: too many AI people, not enough of everything else, eFinancialCareers efinancialcareers.com.
- 7.Talent Power Shift: Hedge Funds in a Hiring Crunch, HedgeCo Insights hedgeco.net.
- 8.Building the Backbone: The Role of Operations in Boutique Managers, HedgeNordic hedgenordic.com.
- 9.The Hidden Margin Drain: Why Manual Data Operations Are Becoming Too Expensive, Institutional Investor institutionalinvestor.com.
- 10.Beyond uncertainty: A modern data architecture unlocks operational excellence in 2025, AIMA aima.org.
- 11.Faster US settlement hit harder than expected, Citi survey says, Reuters reuters.com.
- 12.T+1 Is Looming: What Do Hedge Funds Need to Do to Be Ready?, DTCC dtcc.com.
