OpenAI desperate to avoid explaining why it deleted pirated book datasets

OpenAI is under increasing pressure to clarify its decision to remove two contentious datasets that comprised pirated books, a move that could significantly impact a class-action lawsuit brought by authors claiming their works were used without permission to train ChatGPT. The datasets, referred to as "Books 1" and "Books 2," were deleted before the launch of ChatGPT in 2022. They were originally compiled in 2021 by former OpenAI staff, using data sourced from the open internet, primarily from a controversial digital library known as Library Genesis (LibGen). OpenAI has stated that these datasets became obsolete in the same year they were created, leading to their subsequent removal. However, authors involved in the lawsuit are skeptical of this explanation. They highlight inconsistencies in OpenAI's narrative, noting that the company appeared to retract its claim regarding the datasets' “non-use” as a justification for deletion. This shift occurred after a court allowed the authors to examine OpenAI's internal communications concerning the “non-use” of these datasets. The situation escalated last week when U.S. District Judge Ona Wang mandated that OpenAI provide all correspondence with its legal team regarding the deletion of the datasets, as well as any internal discussions about LibGen that were previously withheld under claims of attorney-client privilege. According to Judge Wang, OpenAI's position that “non-use” was not a reason for deletion contradicted their assertion that it should remain a privileged matter. As the legal proceedings unfold, the authors are keenly awaiting insights into the internal discussions at OpenAI, which could shed light on the true reasons behind the deletion of these datasets.