The early returns on gen AI for software development

Generative AI is already having an impact on multiple areas of IT, most notably in software development.

Early use cases include code generation and documentation, test case generation and test automation, as well as code optimization and refactoring, among others.

Still, gen AI for software development is in the nascent stages, so technology leaders and software teams can expect to encounter bumps in the road. But early returns indicate the technology can provide benefits for the process of creating and enhancing applications, with caveats.

Assistance across the app-dev lifecycle

Gen AI “has opened up the body of knowledge that developers can tap into in a conversation type of paradigm,” says Prasad Ramakrishnan, senior vice president of IT and CIO at Freshworks, a provider of customer service software.

“It almost feels like having a deskside colleague to brainstorm with,” Ramakrishnan says. “Our initial launch of gen AI has enabled our team to develop code in days, instead of weeks.”

The key to success in the software development lifecycle is the quality assurance (QA) and verification process, Ramakrishnan says. “The maturity of any development organization can easily be measured in terms of the size and type of investment made in QA,” he says. “Gen AI is playing a role in assisting with performing code reviews and early detection of potential issues.”

Gen AI is also reducing the time needed to complete testing, via automation, Ramakrishnan says. “Anomaly detection is another area where Gen AI can help identify hidden defects and landmines,” he says.

Financial services firm Vanguard is experimenting “rapidly and safely” with generative AI tools — with human oversight and expertise — enabling productivity gains for developers, says Nitin Tandon, CIO.

“Software and coding development remain a high-value area for experimentation, in addition to content development and knowledge management, in an effort to boost operational efficiencies,” he says.

Early results from a pilot program for gen AI development are encouraging, Tandon says, with developers reporting that gen AI streamlines code generation, debugging, and code consistency.

“Junior developers are reporting the biggest productivity boosts, but this remains an area of active research and experimentation,” Tandon says.

“Additionally, we are looking into training LLMs [large language models] on our code base to unlock further productivity boosts for our developers and data engineers. With our large developer talent base at Vanguard, even a moderate increase in productivity can unlock meaningful value for our clients.”

Financial technology provider Momnt is using Github Copilot, a cloud-based AI tool developed by Microsoft’s GitHub and OpenAI to help users of various development platforms by autocompleting code.

The software development teams at Momnt, including both engineering and QA professionals, use GitHub Copilot to support development of the company’s lending platform, among other tools.

“We want our software developers and engineers to use AI and gen AI tools to help generate various test cases, and our quality assurance engineers will use these tools to conduct evaluations more thoroughly without sacrificing time,” says Brian Lanehart, president and CTO.

“So far, our teams’ collective embrace of new tech solutions and gen AI has been positive,” Lanehart says. “They continue to be curious, seeking out new tools to implement in their day-to-day processes. Doing so not only streamlines daily tasks, but also improves overall team efficiency.”

As development teams become increasingly familiar with these tools, their understanding of the tools and ability to apply them to diverse business scenarios will bring even greater value, Lanehart says.

“For example, being able to completely communicate an entire application request to gen AI that generates all necessary code will reduce a task timeline significantly,” Lanehart says. That means an engineer or team is freed up to spend more time thinking creatively or strategically about the overall project and how to further improve it, he says.

The development team at software company ZoomInfo was eager to experiment with gen AI tools once they became available last year for use at scale, says CTO Ali Dasdan.

One of the earliest use cases provided the company’s software developers with access to Github Copilot. After initial success, ZoomInfo has begun to integrate gen AI throughout its organization to improve productivity, Dasdan says. One example is with document search and summarization.

“Software development requires heavy documentation,” Dasdan says. “Documents such as product requirements and architecture designs are standard in well-run organizations,” but development teams need a lot of time to review these documents. “Gen AI has freed up a significant amount of time by summarizing and indexing these documents in just a few minutes,” he says.

The success of the trial led the company to get licenses for nearly all its software developers. “Our engineers still have to review the code the tool creates, however,” Dasdan says. “We’ve already accepted tens of thousands of lines of code and we’ve realized a significant amount of time saved.”

Limits, team impacts, and lessons learned

One of the key takeaways from early use of gen AI is that it won’t replace human developers.

At IT services provider BDO Digital, initially there was a “wave of excitement” about gen AI’s potential to autonomously generate complex software, says Kirstie Tiernan, principal in the firm’s data and AI practice.

“However, we quickly learned that AI is a tool to augment human expertise, not replace it,” Tiernan says. “The need for human oversight to ensure the quality and functionality of AI-generated code quickly became apparent. It’s a partnership where AI handles some of the heavy lifting, allowing developers to focus on strategic problem-solving.”

A key lesson BDO is working through is the importance of integrating AI tools with existing workflows. “It isn’t just about adopting new tools; [it’s] more about how development teams operate, communicate, and collaborate,” Tiernan says. “The integration process highlights the need for flexibility and adaptability in all of our development practices.”

One of the more interesting surprises at BDO was the impact of gen AI on creativity and innovation. “By automating routine tasks, developers have been freed up to tackle more complex challenges and explore more innovative solutions,” Tiernan says. “It’s exciting to see how AI can serve as a catalyst for human creativity and ideation.”

With any new technology solution, one of the biggest challenges is identifying the extent to which a team should integrate or rely on the tool, Momnt’s Lanehart says.

“One of our core beliefs is using technology to empower and support people,” he says. “So, we knew that we did not want AI and gen AI tools to function as a replacement for our employees. Instead, we wanted the tools to complement the skills that these individuals bring to the team and help them function more effectively and efficiently.”

The technology industry overall is seeing more demand for people who can oversee, implement, and run gen AI tools, Lanehart says. For example, this could be an employee who functions on the product development team, but whose core expertise is ChatGPT or Copilot, he says.

As gen AI becomes increasingly prevalent, “we’re seeing value in having people who are cross-trained with these tools,” Lanehart says. “We want people who can solve new problems in diverse ways, and we want them to bring that knowledge back to our team.”

Momnt has begun encouraging its software development team members to expand their understanding of gen AI tools by applying them to their personal interests, such as music, comedy, and other areas, Lanehart says. “Finding overlaps between the applications of AI to both fintech and personal interest puts our team in a unique position to drive new industry growth,” he says.

Lyric, a healthcare technology company, is harnessing the power of LLMs to improve several processes, says Akshay Sharma, chief AI officer. But one of the early lessons was how much work was needed to get the correct value from LLMs.

“Out of the box they are somewhat generalized, miss the mark, and hallucinate,” Sharma says. “But, with the right engineering and design [and by] running experiments with prompts, we get a lot of mileage out of this. We had to build a lot of experimental and testing frameworks to continuously evaluate gen AI.”

Freshworks’ Ramakrishnan believes Gen AI has the potential to enable developers to bring applications to market faster; “however, their skillset will need to adjust to be professionals at prompt engineering,” he says. “AI-generated coding will only be as valuable and accurate as the type of prompt that is asked.”

Moreover, AI code needs to be verified by experienced developers to confirm accuracy, Ramakrishnan adds. “I can’t overemphasize the value of code reviews by humans on machine-generated codes,” he says. “Despite its productivity value to the workplace, AI is far from perfect and requires oversight.”

In addition, the use of AI raises some ethical issues related to the introduction of bias in algorithms, which can lead to unintended consequences if not checked, Ramakrishnan says. “It also introduces new considerations in the area of information security,” he says. “Bad actors now have a broader reach for introducing malicious code in millions if not billions of systems.”

There will be a constant need to re-tool the workforce to make effective use of AI, Ramakrishnan. “This said, we are barely scratching the surface of gen AI’s productivity value,” he says. “The best days are ahead of us.”

Taking a developers’-eye view

One of the best ways to determine the impact of gen AI on development teams is to ask team members to weigh in on their experiences. To collect internal feedback on ZoomInfo’s use of GitHub Copilot, the company conducted a survey of about 80 of its developers. The research showed that Copilot has several strengths.

One is the ability to generate boilerplate and repetitive code, which enables developers to focus on complex logic. Another is a drastic reduction in the time it takes to write unit tests. “Many users report that the tool improves their coding speed by offering useful code suggestions and auto-completing lines,” ZoomInfo’s Dasdan says.

These strengths resulted in several benefits for ZoomInfo’s developers, with a large majority saying Copilot reduced the amount of time it takes to complete tasks, by an average of 20%. About two thirds said using the gen AI technology allowed them to complete more tasks per sprint, and about three quarters said the quality of their work was improved.

“Based on these strong early results, we anticipate gen AI tools continuing to improve the productivity of our engineers, as well as save time from tasks that are auxiliary to writing actual production code,” Dasdan says.

Software development’s gen AI future

Development leaders are confident that gen AI will only grow in importance as a development tool.

“Looking ahead, the potential for productivity gains with generative AI is substantial,” BDO Digital’s Tiernan says. “As these tools become more integrated into the fabric of software development, we’re likely to see dramatic reductions in development time and costs.”

For example, automating the generation of boilerplate code and providing real-time suggestions for bug fixes can halve the time traditionally needed for certain development tasks, Tiernan says.

“But the real game changer will be in how generative AI enables us to tackle more complex problems more efficiently,” Tiernan says. “With AI handling routine aspects, developers can focus on strategic innovation, pushing the boundaries of what’s possible in software solutions.”