Coinbase Tests Token Security with ChatGPT
- Crypto exchange Coinbase recently used the AI chatbot ChatGPT to conduct token security tests on 20 smart contracts to determine their accuracy.
- Out of 20, 12 times ChatGPT produced the same result as a manual review, with the promising speed of conducting a smart contract analysis.
- Of the 8 misses, 5 of them were cases where ChatGPT incorrectly labeled a high-risk asset as low risk.
- The exchange said that ChatGPT can be inconsistent, i.e., when asked the same question multiple times, different answers are given by the AI chatbot.
ChatGPT, an artificial intelligence (AI) chatbot developed by OpenAI and launched in November 2022, has become very popular as people figure out various ways to make use of the platform to make their tasks easier. Interestingly, many AI enthusiasts believe that ChatGPT can replace manual labor, but the largest crypto exchange in the United States, Coinbase, recently proved that this is not the case by testing token security with the chatbot and revealing the results to the public.
According to a blog post from the crypto exchange on March 20, Coinbase used ChatGPT to perform automated token reviews via its ERC20 token review framework and compared the results to standard token security reviews. The result was clear: ChatGPT did not meet the accuracy standards set by the firm to allow the chatbot to be integrated into the exchange’s security arsenal.
By comparing the ChatGPT and manual security reviews side-by-side, Coinbase confirmed that the manual security reviews are more accurate. The exchange compared the security reviews of 20 smart contracts, and it is crucial to note that ChatGPT produced the same result 12 times as the manual review.
However, of the 8 misses, 5 were cases where ChatGPT incorrectly labeled a high-risk asset as low-risk. The exchange noted that this was a severe failure since “underestimating a risk score is far more detrimental than overestimating.”
Coinbase noted that ChatGPT displays a promising speed of smart contract analysis as compared to manual review, which takes more time. However, it doesn’t meet the accuracy standards of a manual review.
“While the efficiency of a ChatGPT review is remarkable, there are still some limitations that impair the tool’s accuracy. Primarily, ChatGPT is not capable of recognizing when it lacks context to perform a robust security analysis. This results in coverage gaps where additional dependencies go unreviewed,” said Coinbase.
Coinbase also added that ChatGPT can be inconsistent, i.e., when asked the same question multiple times, different answers are given by the AI chatbot. The exchange added that ChatGPT is also “influenced by comments in the code and seemed to occasionally default to comments rather than function logic.”
“We anticipate that with further prompt engineering, we can drive the accuracy of the tool up. However, the tool cannot be solely relied upon to perform a security review,” Tom Ryan, the Blockchain Security Engineer at the crypto exchange.
According to the crypto exchange, depending on ChatGPT for security reviews would be a costly misstep, but one can increase accuracy by using the AI chatbot in combination with manual review to catch any risk that may have gone overlooked.
As reported earlier by BitcoinWisdom, Coinbase confirmed that it had an exposure of around $240 million to Signature Bank, a New York-based banking institution that was closed by the New York Department of Financial Services (NYDFS) earlier this month. The Federal Deposit Insurance Corporation (FDIC) was established as the receiver of the bank.
On the other hand, the crypto exchange also plans to expand outside the United States, which can be seen as an attempt to compete with Binance, the world’s largest crypto exchange by trading volume. A Coinbase spokesperson wrote in an email that the company’s international expansion drive will “focus on high-bar regulatory jurisdictions,” while adding that the exchange remains “focused on offering our products in a safe, compliant way, with sound risk management at the core.”
A crypto journalist with over 3 years of experience in DeFi, NFT, metaverse, etc. Parth has worked with major media outlets in the crypto and finance world and has gained experience and expertise in crypto culture after surviving bear and bull markets over the years.
