There is no disputing that the AI still has a few unreliable moments, but one can hope that at least its tests will be accurate. However, last week Google allegedly instructed contract workers testing Gemini not to leak any information, regardless of who they are, TechCrunch reports based on the internal guidelines it viewed. Google shared a preview of Gemini 2.0 earlier this month.
Google has reportedly instructed GlobalLogic, a company that provides external services to contractors that test AI-generated products, not to have reviewers pass information outside of their expertise. In the past, contractors could choose to skip any warning that didn’t pass their standards – such as asking a doctor about the rules. The guidelines stated, “If you do not have the necessary expertise (eg coding, math) to rate this information, please skip this activity.”
Now, contractors are said to be instructed, “You must not skip information that requires domain-specific knowledge” and that they must “limit the parts of the information you understand” while adding a note that is not their area of expertise. Obviously, the only times contracts can now be breached is if a large part of the information is missing or contains dangerous content that requires certain consent forms to be scanned.
One contractor responded positively to the changes saying, “I thought the point of the jump was to increase accuracy by giving someone a better one?”
Shortly after this article was first published, Google provided Engadget with the following statement: “Associates perform many tasks in many Google products and platforms. They provide valuable feedback not only on the content of the answers, but also on the style, format, and other aspects. The ratings they provide do not directly affect our algorithms, but when combined, they are a useful data point to help us measure that our systems how well they work.”
The spokesperson for Google also noted that the new language should not necessarily lead to a change in the accuracy of Gemini, because they ask partners to rate the parts of the instructions that they understand. This may be to provide feedback on things like formatting issues even if the reviewer has no specific expertise in the topic. The company also pointed to this week’s release of the FACTS Grounding benchmark that can check LLM responses to ensure that “that is not only true about the input provided, but also has enough detail to provide satisfactory answers to user questions.”
Update, December 19 2024, 11:23AM ET: This story has been updated with a statement from Google and more details about how its rating system works.
Source link
