Tracking efficiency gains vis-a-vis story points.

Measuring efficiency gain is hard.

Efficiency gains are not realized immediately. In fact, there's almost always a drop in efficiency when a process is changed or a new tool is introduced. People need time to learn and understand the new tool or method and become comfortable with it as part of their way of working. Sometimes, these decreases may be very small, but they are usually there. There needs to be sufficient time between adopting the change and verifying that the change had the desired effects.

Estimates are not a good way to measure efficiency. Estimates are made early, when little is known about the work. By doing the work, the team could realize that their estimates were ov.er or under the actual effort or complexity. Basing evaluations on actual data that reflects the work completed is likely to be more useful. This is on top of the problem of slight variations in even similar work items and changes in the team that may compound to create differences between historical data.

My preference is to look at how the team is spending their time:

Is the team increasing their time spent on work that is perceived to be valuable (e.g., delivering new features or functionality as opposed to fixing bugs)? This must acknowledge, though, that ongoing maintenance work (such as updating tools and dependencies) exists, so there will never be a 100% allocation to new features.

This doesn't necessarily address whether that work is truly valuable. However, realizing that lags. You often don't know if what you built was truly valuable until you make it available to stakeholders and can observe whether they are using it and if they report any problems (or ideas for improvement) with the delivered capabilities. If the team is delivering work that isn't used or isn't well-received, that would point to issues with discovery and definition rather than development efficiency.

Thanks Thomas.  I like the idea of tracking changes in where time is spent.

The question is: what approach do you take to making your estimates in terms of story points? Do you focus more on effort or complexity when making your estimates?

If you focus on effort, the same story should have fewer story points after efficiency optimisation than before. In that case, success cannot be measured by higher velocity, but instead by lower estimations without decreasing velocity. However, you would also need to revise all existing estimates in the backlog downwards.

It's different if you estimate complexity. In my opinion, the complexity of a task is independent of the methods or tools used to undertake it. Methods and tools do not change the task itself, only my ability to complete it.
The team(!) can then assess the success of a measure to increase efficiency by looking at the increase in velocity.

However, if management requires evidence that the team is becoming more efficient, this could be risky. As is well known, the team could simply provide higher estimates, either consciously or unconsciously, which would make the 'measured efficiency' increase.
It might be possible to evaluate a single efficiency optimisation if the company culture does not penalise the team for unsuccessful optimisations.

Ultimately, management should ask itself whether the team is becoming more effective by delivering more value per unit of time. Even an increase in efficiency must ultimately lead to an increase in effectiveness.
In my opinion, value is sometimes hard to measure, but normally harder to fake than efficiency.

As new tools/methods become available, higher-ups want to see some quantified measure of efficiency gains for the teams.

Have the constraining factors proven to be technical rather than cultural?

For the reasons you mentioned, story points are not a measure of performance. They are a planning tool, a way for the team to estimate and forecast capacity within a Sprint. 

Performance measurement in Scrum and Agile is not straightforward, because work is completed collectively as a team, with a focus on value delivered rather than individual output or speed.

Scrum and Agile are also about more than throughput. Transparency, stakeholder collaboration, and early delivery of usable functionality may even be preferred by stakeholders over pure “delivery speed.” It helps to understand what characteristics stakeholders truly value.

Some ideas for communicating performance include focusing on value, quality, and flow:

• Cumulative Flow Diagram (CFD): Visualises flow stability and identifies bottlenecks.
• Cycle Time / Lead Time: Measures how long it takes for work to move from “in progress” to “done.”
• Escaped Defects: The number of defects found in production, a quality indicator.
• Rework Rate: Frequency of stories reopened or changed after completion.
• Stakeholder Satisfaction: Qualitative feedback on perceived value and responsiveness.
• Team Health Metrics: Team engagement, morale, and psychological safety.
• Predictability: How consistently the team meets its Sprint goals or delivery forecasts.

These provide a far more accurate picture of improvement over time than velocity or story points ever could.