Microsoft unveiled its new MInference technology on the AI platform Hugging Face, showcasing a potential breakthrough in processing speed for large language models.

The demonstration, powered by Gradio, allows developers and researchers to test Microsoft’s latest advancement in handling lengthy text inputs for AI systems directly in their web browsers. This interactive demo highlights significant improvements in the efficiency and speed of processing long text inputs, which is a critical aspect of advancing AI technology.

MInference, which stands for “Million-Tokens Prompt Inference,” aims to significantly accelerate the pre-filling stage of language model processing, a common bottleneck for long text inputs.

According to Microsoft researchers, MInference can reduce processing time by up to 90% for inputs of one million tokens, equivalent to about 700 pages of text, while maintaining accuracy. This dramatic reduction in processing time addresses a significant challenge in deploying large language models widely.

The computational demands of large language model (LLM) inference are a substantial barrier, especially as prompt lengths increase. The attention to computation complexity often results in lengthy processing times.

For example, it takes 30 minutes for an 8 billion parameter LLM to process a prompt of one million tokens on a single Nvidia A100 GPU. MInference reduces this inference latency by up to 10 times for pre-filling on an A100 GPU, while maintaining accuracy, showing an 8.0x latency speedup for processing 776,000 tokens.

This innovative approach addresses the AI industry’s need for efficiently processing larger datasets and longer text inputs. As language models grow in size and capability, handling extensive context becomes crucial for applications ranging from document analysis to conversational AI.

The Gradio-powered demo enables developers to explore AI acceleration hands-on, potentially accelerating the refinement and adoption of MInference technology.

Beyond speed improvements, MInference’s ability to selectively process parts of long text inputs raises questions about information retention and potential biases. The technology’s approach to dynamic sparse attention might also reduce the energy consumption required for processing long texts, aligning with concerns about the carbon footprint of AI systems.

This could influence future research directions toward more sustainable AI technologies. Microsoft’s public demo of MInference intensifies competition among tech giants, potentially prompting rapid advancements in efficient AI processing techniques.

The coming months will likely see intense scrutiny and testing of MInference, providing valuable insights into its real-world performance and implications for the future of AI.

Meta AI researchers have introduced MobileLLM, a new way to make efficient language models for smartphones and other devices with limited resources. Published on June 27, 2024, this research challenges the idea that powerful AI models need to be very large.

The team, including experts from Meta Reality Labs, PyTorch, and Meta AI Research (FAIR), focused on creating models with fewer than 1 billion parameters, much smaller than models like GPT-4, which have over a trillion parameters.

The research team made significant innovations in MobileLLM by prioritizing model depth over width, implementing embedding sharing, grouped-query attention, and a novel block-wise weight-sharing technique.

These strategies allowed MobileLLM to outperform previous models of similar size by 2.7% to 4.3% on common benchmark tasks. Though the improvements seem minor, they are notable in the competitive field of language model development.

Yann LeCun, Meta’s Chief AI Scientist, highlighted these advancements on social media, underscoring the importance of the team’s work.

A remarkable achievement of MobileLLM is its 350 million parameter version, which demonstrated comparable accuracy to the much larger 7 billion parameter LLaMA-2 model in certain API calling tasks.

This finding suggests that more compact models could offer similar functionality to larger models while requiring significantly fewer computational resources. The ability to achieve high accuracy with smaller models is particularly valuable for applications on resource-constrained devices.

The development of MobileLLM reflects a broader trend towards more efficient AI models. As advancements in large language models begin to plateau, researchers are increasingly focusing on the potential of smaller, specialized designs.

MobileLLM’s efficiency and suitability for on-device deployment place it in the emerging category of Small Language Models (SLMs). Despite its name, MobileLLM aligns with this movement towards compact AI models that can perform effectively without massive computational power.

Although MobileLLM is not yet available for public use, Meta has open-sourced the pre-training code, enabling other researchers to build upon this work. This initiative may eventually lead to more advanced AI features on personal devices, though the timeline and specific capabilities are still uncertain.

The development of MobileLLM marks a significant step towards making advanced AI more accessible and sustainable, challenging the notion that only large-scale models can be effective and opening new possibilities for AI applications on personal devices.

Researchers at Osaka Metropolitan University have developed an artificial intelligence tool that can assess lung function by analyzing chest X-rays, a breakthrough that could revolutionize how we monitor respiratory health.

Traditionally, doctors use spirometry, which requires patients to blow into a device and follow specific instructions. However, this can be challenging for certain groups, like infants or those with cognitive impairments.

The AI model created by Associate Professor Daiju Ueda and Professor Yukio Miki was trained on a massive dataset of more than 140,000 chest X-rays collected over nearly two decades.

They compared the AI’s predictions with actual spirometry results and found a strong correlation, indicating its ability to accurately estimate lung function without the need for physical exertion from the patient.

This innovation addresses significant challenges in current healthcare practices by offering a non-invasive alternative for assessing pulmonary function. By relying solely on existing chest X-rays, the AI model simplifies the diagnostic process and potentially reduces medical costs associated with more complex tests.

Professor Ueda emphasized the collaborative effort involved in developing this AI model, which included contributions from healthcare professionals, researchers, and patients across various institutions.

He expressed optimism about the tool’s potential to alleviate patient burdens and improve healthcare efficiency by providing accessible and accurate lung function assessments.

Brazil’s National Data Protection Agency (ANPD) has taken decisive action against Meta, suspending its new privacy policy that permits the use of user content for training AI models.

This move follows concerns over potential harm to users’ fundamental rights, prompting ANPD to issue a five-day ultimatum for Meta to revise its policy or face substantial daily fines.

Similar to actions taken in Europe, where Meta’s plans were halted due to regulatory concerns and complaints, Brazil’s ANPD is asserting its authority to protect user data and privacy. Meta, however, asserts that its policy changes comply with local laws and expresses disappointment over the setback to innovation and AI development in Brazil.

The situation in Brazil differs significantly from that in Europe due to several key factors. Notably, Meta’s policy changes in Brazil affect users of all ages, including teenagers and children, which raises legal and ethical concerns under Brazilian laws. In contrast, Europe’s objections primarily focus on adults, reflecting a narrower scope of regulatory concern.

Moreover, the process for users to opt out of Meta’s AI training data collection is more cumbersome in Brazil compared to Europe. Brazilian users face a complex eight-step procedure, whereas Europeans have a simpler form-based opt-out option.

This disparity underscores challenges in enforcing data privacy rights across different jurisdictions and suggests Meta’s varying approach in compliance.

Critics, including Data Privacy Brasil’s Pedro Martins, highlight these discrepancies as indicative of Meta’s potentially dismissive stance towards Brazilian regulations. The ANPD’s intervention signals a push for Meta to align its policies with local standards and may necessitate significant revisions before implementation can proceed in Brazil.

AI agents are emerging as a promising research area with real-world applications. These agents use advanced models like large language models (LLMs) and vision language models (VLMs) to understand natural language instructions and achieve complex goals on their own.

They use tools like browsers, search engines, and code compilers to check their actions and think about their tasks. However, researchers at Princeton University have found several problems with current benchmarks and evaluation methods, which limit the practical use of AI agents.

One major issue highlighted is the lack of cost control in agent evaluations. AI agents often require more computational resources than single-model calls, making them expensive to run.

To enhance accuracy, some systems generate multiple responses and use voting or verification mechanisms to select the best one, which significantly increases computational costs.

While maximizing accuracy is feasible in research settings, practical applications must consider budget constraints for each query. The researchers suggest visualizing evaluation results as a Pareto curve of accuracy and inference cost, advocating for joint optimization of these metrics to develop cost-effective and accurate agents.

The researchers evaluated different prompting techniques and agentic patterns, discovering substantial cost differences for similar accuracy levels. They argue that joint optimization can lead to less expensive yet accurate agents, enabling researchers to balance fixed and variable costs effectively.

For instance, investing in optimizing the agent’s design can reduce the variable cost by using fewer in-context learning examples. Testing this approach on the HotpotQA benchmark demonstrated that joint optimization could achieve an optimal balance between accuracy and inference costs.

Another critical issue is the difference between evaluating models for research purposes and developing real-world applications. In research, accuracy is often prioritized, with little regard for inference costs.

However, inference costs are crucial for practical applications, influencing model and technique choices. Evaluating inference costs is challenging due to variable pricing by model providers and fluctuating API call costs.

To address this, the researchers created a website that adjusts model comparisons based on token pricing and conducted a case study on NovelQA, revealing that benchmarks for model evaluation can be misleading for downstream applications.

Overfitting is a significant problem in AI agent benchmarks, as models often find shortcuts to perform well on benchmarks without real-world applicability. This issue is exacerbated by the small size of agent benchmarks, typically only a few hundred samples.

The researchers recommend that benchmark developers create and maintain holdout test sets with examples that cannot be memorized during training and can only be solved by understanding the task. Their analysis of 17 benchmarks revealed that many lacked proper holdout datasets, allowing agents to take unintended shortcuts.

The researchers tested WebArena, a benchmark for evaluating AI agents on different websites, and found several shortcuts in the training datasets. These shortcuts allowed agents to overfit tasks, leading to inflated accuracy estimates and over-optimism about agent capabilities.

They emphasize that AI agent benchmarking is a new field without established best practices, making it challenging to differentiate genuine advancements from hype.

The researchers argue that benchmarking practices need to be rethought to accurately assess the capabilities of AI agents, which could soon play a significant role in everyday applications.

After facing multiple rejections, Apple has approved Epic Games’ third-party app marketplace for launch in the EU under the EU’s Digital Markets Act (DMA).

Earlier this year, Epic announced plans to bring its digital storefront and Fortnite back to iOS in Europe, but Apple initially rejected the submission twice. Apple cited concerns that Epic’s Games Store looked too similar to Apple’s App Store, prompting Epic to escalate the issue to European regulators.

Later on the same day, Apple conditionally approved the marketplace app, requiring Epic to make specific adjustments in a future update. The initial rejections were based on details like the position of the “Install” button and the “in-app purchase” label, which Apple found too similar to its own.

Epic argued that these naming conventions are standard across popular app stores and criticized Apple’s actions as arbitrary and obstructive, claiming they violated the DMA.

Apple stated that it had approved the Epic Sweden AB Marketplace app but emphasized that Epic must still address the required changes.

Under Apple’s developer agreement, Epic agreed not to make its Game Store appear too similar to Apple’s App Store, but Apple found issues with the design and labeling of the download button. Despite these concerns, Apple noted that Fortnite’s app had already been approved.

This situation illustrates Apple’s strict enforcement of new rules permitting third-party app stores on iOS, potentially discouraging other developers from establishing their own distribution channels.

The DMA requires Apple to allow third-party app stores on the iPhone, as it is classified as a “gatekeeper” under EU law. Developers can also adopt new DMA rules to reduce commissions on in-app purchases, although the process involves a complex fee structure for using Apple’s technology.

Epic Games has long sought a way to reach iPhone customers without paying commissions on in-app purchases, leading to an extended legal battle with Apple. Despite losing most of its claims in U.S. courts, Epic continued to challenge Apple’s policies.

After Apple briefly terminated Epic’s developer account in the EU, regulators intervened. Epic remains committed to launching Fortnite on iOS and iPad in the EU and bringing its Game Store to the market, confirming the approval of its marketplace app on social media and to TechCrunch.

Japan’s Digital Agency recently achieved a milestone by eliminating the use of floppy disks in its government computer systems. While floppy disks are largely considered obsolete in today’s technological landscape, they are still being utilized in Japan for essential functions, notably in an environmental system monitoring vehicle recycling.

This move marks a significant step forward in Japan’s efforts to modernize its digital infrastructure.

Digital Minister Taro Kono proudly declared the achievement on June 28, emphasizing the symbolic end of an era. Since his appointment to the Digital Agency in 2022, Kono has been proactive in pushing for the abandonment of outdated technology across approximately 1,900 government procedures.

This initiative included phasing out not just floppy disks but also fax machines, CDs, and MiniDiscs, which were remnants of ’90s-era technology.

Kono’s campaign against obsolete technology garnered attention not just domestically but also internationally, underscoring the broader trend of countries grappling with legacy systems long past their prime.

The United States military, for instance, famously continued to use 8-inch floppy disks until 2019 to operate the Strategic Automated Command and Control System (SACCS), responsible for nuclear command and control. This revelation, highlighted by CBS’s 60 Minutes, brought to light the persistence of outdated technology in critical infrastructures worldwide.

Japan’s transition away from floppy disks reflects a broader global effort to modernize digital infrastructure and enhance operational efficiency. The elimination of these antiquated technologies is seen as a necessary step to streamline governmental processes and ensure compatibility with contemporary data storage and communication methods.

As countries like Japan continue to embrace digital transformation, the focus remains on updating systems to meet the demands of a rapidly evolving technological landscape.

Pixel Perfect Dude unveiled new details about their upcoming game, #Drive Rally, providing a deep dive into its gameplay and features. Unlike its predecessor, the endless racer #Drive, #Drive Rally introduces significant enhancements.

Central to the game are its 36 cars spanning three distinct classes, featuring both returning favorites like Das Holzwagen and The Doggo from #Drive, alongside newcomers like Das Sandsturm and Celestia.

Each vehicle offers extensive customization options, allowing players to tailor their cars to suit their preferences. Additionally, every car includes a co-driver who not only navigates but also offers commentary on the player’s driving skills, adding a dynamic layer to the gameplay experience.

Set against the backdrop of the 1990s, a departure from #Drive’s 1970s and 1980s setting, #Drive Rally promises a nostalgic journey with numerous ’90s-themed Easter eggs sprinkled throughout the game.

The environments are diverse, ranging from serene German forests to the rugged deserts of the American Southwest, ensuring a varied and immersive racing experience. Pixel Perfect Dude has announced that the game will debut in early access on PC and Mac during autumn, with plans for a console release slated for 2025.

Fans eagerly anticipate #Drive Rally’s release, drawn by its expanded car roster, detailed customization features, nostalgic setting, and varied racing environments.

Pixel Perfect Dude’s commitment to enhancing the gameplay experience with interactive co-drivers and thematic elements from the 1990s sets #Drive Rally apart as a promising addition to the racing game genre, promising both innovation and nostalgia for players worldwide.

Salesforce has introduced the xLAM-1B model, a compact AI with just 1 billion parameters that surpasses much larger models in function-calling tasks. Referred to as the “Tiny Giant,” this innovation challenges the conventional belief that bigger models inherently perform better.

Salesforce’s success stems from its APIGen pipeline, which curates high-quality datasets for training, focusing on diversity and verifiability across 3,673 APIs in 21 categories.

In performance benchmarks, xLAM-1B outshines models seven times its size, including industry benchmarks like GPT-3.5 and Claude. This achievement underscores the efficiency gains possible through meticulous data curation over sheer model scale, suggesting a paradigm shift in AI development strategy.

By prioritizing data quality, Salesforce demonstrates that smaller, more resource-efficient models can compete effectively while potentially reducing computational demands and environmental impact.

The implications extend beyond technical prowess to practical applications. On-device AI capabilities, once constrained by the need for cloud computing, could now flourish. xLAM-1B’s ability to execute complex tasks locally on devices promises faster response times and enhanced privacy, addressing critical concerns in the era of data sovereignty.

This development aligns with broader trends towards edge computing and IoT, paving the way for a more distributed AI ecosystem.

Public availability of Salesforce’s dataset of 60,000 function-calling examples further accelerates innovation by inviting collaboration and validation from the wider research community. This openness fosters advancements in AI tailored for specific applications, potentially democratizing access to sophisticated AI capabilities beyond tech giants.

CEO Marc Benioff heralds this achievement as a milestone in “on-device agentic AI,” envisioning a future where advanced AI services are integrated seamlessly into everyday devices. This vision not only challenges existing norms but also redefines the future landscape of AI, emphasizing efficiency, responsiveness, and accessibility.

Salesforce’s xLAM-1B represents a significant breakthrough in AI, showcasing how meticulous data curation and efficient model design can eclipse the performance of larger counterparts.

This accomplishment not only sets a new standard for AI efficiency but also unlocks possibilities for widespread adoption of on-device AI applications, heralding a transformative era in artificial intelligence.