A senior manager in our test group down at Eglin Air Force Base forwarded me a white paper by SOGETI a company in Europe. The white paper was titled, “Testing of Artificial Intelligence, AI Quality Engineering Skills – An Introduction”. In general the white paper was well done however the authors based their conclusion that AI testers will need a plethora of new skills to be able to test AI on the following statement, “AI is not required to learn, it could be using pre-programmed rules to handle all possible outcomes. However, for systems with more than basic complexity, this has proved to be a task too large and too complex to handle (it has been tried and failed multiple times since the 1960s)”. What these authors apparently didn’t now is that BRC did solve this problem in 1996 under the Army’s small business innovation research (SBIR) program. In 1998 BRC was awarded the Tibbetts award from the small business administration for “excellence in innovation” for this breakthrough.

Of course it went largely unnoticed by most Department of Defense contractors and customers because in 1998 nobody cared about artificial Intelligence or Machine Learning (AI/ML). There was little, if any money in AI/ML except for a few small programs here and there. Even DAPRA who is charged with developing next-generation technologies for the DoD largely ignored these technologies.

Enter the resurgence in the interest in AI/ML a few years ago as a result of Elon Musk, GOOGLE, IBM, and a few others. The real interest within the DoD came when the Chinese Government started a robust AI/ML development program. For years the DoD stated emphatically that they would not give control of weapons systems to an artificial intelligence, after the Chinese started their program and commercial companies in Europe started testing self-driving cars it finally became evident that the U.S. had to enter the race or be left behind.

Overnight “the HERD”…(I’ve spoken about them before) appeared. The HERD is a group of companies that thunder from hot topic to hot topic claiming expertise in an area that is receiving increasing funding from congress. The same companies that literally overnight claimed to be “IT companies” 20 years ago, just 5 years ago were claiming to be “CYBER companies” and all of a sudden are now “AI companies”. But I digress….

There is a solution to the AI testing problem that fits extremely well with the way that the DoD does business in the first place. The BRC CORE toolset is based upon the rapid creation of finite state machines that are so complex I sometimes brief them as “infinite state machines”. Despite the claim by the SOGETI authors that preprogrammed rules (states) could not be done, it has been done and its quite easy. This method has been the basis of the CORE tools since they were developed in 1996 and they have proven to be extremely valuable at creating complex AI (Cognitive) algorithms for many special projects that BRC has done for DOD and Other Defense Agencies (ODA) since that time. Just to give a few examples BRC was the first company to run and demonstrate 150K intelligent entities in near-real time in a SPEEDES-Based simulation at the Institute for Defense Analysis (IDA) in about 2000. The next year we were in the Human Behavioral Testbed Challenge from NAWCTSD where the CORE infinite state machine was declared the winner after it clearly outperformed both large companies and universities all using the same scenario. Our intelligent entities were so robust they have been used in ModSAF, ONESAF, JSAF, SOFSAF, DICE, DIADS, ITEMS, SPEEDES, WARSIM and EADTB. This technique has also been used to build MTI trackers, cognitive mentors, battle damage assessment tools, and recently a unique compressed bandwidth communications system algorithm that promises to change the way that the DOD communicates on the modern battlefield.

So how do you make testing artificial intelligence / Machine Learning easy? First you don’t allow the HERD to implement old outdated algorithms in military systems. If the test community allows HERD companies to implement algorithms that have no known hope of ever being validated into military systems then its their own fault when an unmanned vehicle running Bayesian network-based autonomy algorithms puts a rocket squarely through the side of a middle school in downtown Crestview near Eglin AFB. Second and most importantly there is no need to use these fun (but useless and expensive) approaches when a solution exists that pre-validates the input and output data during development. Using the CORE approach will allow AI-based algorithms to be validated using the same skills and techniques that the test community has employed for the last 50 years.

Some countries don’t care if they lose a middle school or two along the way…its just the price of progress to them. However in the United States we have both a moral and a legal responsibility to do the right thing. I would strongly encourage those decision-makers within the DOD to stop “riding the HERD” and realize that we have an existing method to create AI-based algorithms that can easily be validated during testing. In fact last year one of my colleagues at BRC implemented an idea I had 20 years ago but never had time to follow-up on and created a new CORE tool that can create learning algorithms that can be fully validated in much the same way.

As I calmly step down from my soapbox, wiping the sweat off my furrowed brow I have to shake my head and frown. I already know that one voice cannot overcome the power of the HERD. I have been a lone voice in the wilderness for almost 30 years talking about why the Government “DOES NOT WANT AI IN ITS SYSTEMS”. To implement a technology in a dangerous military system when it is well-known that its output cannot EVER be adequately validated, and when there is an alternative approach already available that works just as well or better (in my humble opinion) is both immoral and criminal.

Dr. Andy

BRC CEO and CTO

“Cognitive Psychophysicist with an interest in almost everything”

“40 years experience working in AI for the DOD”

FOOTNOTE: It was suggested to me by a very astute employee that I add a sentence or two each time I add an article to the blog to highlight one or more of our most pressing current open hiring requisitions. These can be found on the BRC website at www.brc2.com. BRC Employees that refer someone that is hired to fill one of these open positions will receive a minimum referral bonus of $1000 (after tax). Here is today’s featured open position:

Subject Matter Expert 28 TES/DOW

Job Code: 18-PTO-022
Location: Eglin AFB, FL
Clearance Required: Secret

Job Description/Essential Functions:

Bevilacqua Research Corporation (BRC) is a Service Disabled Veteran Owned, Small Business based in Huntsville, Alabama incorporated in 1992. BRC provides “best-value” technical and management solutions to a diverse government and commercial customer base in the following key business areas: test & evaluation, systems analysis & integration, security & intelligence, and R&D.

Individual provides expert level support of Program Manager and Test Director duties during the planning, execution, and reporting phases of each 28th Test and Evaluation Squadron developmental and operational test.

Responsibilities:

• Provides expert level support of Program Manager and Test Director duties during the planning, execution, and reporting phases of each 28 TES developmental test (DT) support and MAJCOM operational test (OT).
• Assist with detailed understanding of DoD test organizations through recent test aircrew experience in laboratory, modeling and simulations, ground and flight test processes and procedures, flight and ground safety, operational risk management, resource development/planning, test team management, environmental planning, and execution and reporting of laboratory/modeling and simulations/ground/flight test operations.
• Coordinate unique test platforms to access airspace, ranges and other test facilities in order to effectively conduct laboratory, modeling and simulation, ground and flight test operations.
• Assist leadership in the identification of test objectives, methods of testing, test support and documentation requirements for test projects.
• Assist in the analysis of capabilities of new B-2 hardware and software and the evaluation of impacts of new technology on current systems and policies.
• Assist in the evaluation of advanced aerospace technologies, electronic warfare system performance improvements, technology demonstrations and fielding schedules using recent test experience.
• Provide assistance with analysis, risk mitigations and alternatives in DT and OT of integrated aircraft systems/subsystems to ensure the government receives a functional and fully supportable product for the combat crews.
• Assists with expert in-depth and recent knowledge of, and familiarity with, aircraft systems, flight operations, security guidance, data processing software capabilities, and USAF policies to provide expert advice to 28 TES government personnel.
• Experience testing on a variety of electronic warfare system laboratories, models and simulations and open air ranges to include, but not inclusive, Eglin AFB Multi-spectral Test and Training Environment, NAS China Lake Electronic Combat Range, Nellis AFB Test and Training Range, Utah Test and Training Range and NAS Point Mugu.

Minimum Requirements:

• U.S. Citizenship is required
• Active SECRET clearance is required, with ability to obtain a TS/SCI clearance; investigation within the last five years
• Must be able to provide expert advice and assistance in technical, engineering, and integration support in the operational test and evaluation of bomber aircraft electronic warfare systems, sensors, and weapon systems.
• Must be able to provide expert support for programs involving aircraft compatibility testing.
• Bachelor’s degree required.
• Minimum of 8 years flying experience as pilot, Weapons Systems Officer, or Electronic Warfare Officer in B-2, B-1B, B-52, EC-130, RC-135 with less than 5 years removed from active flying

Other Job Information (if applicable):

• Minimum of 8 years as a B-2 pilot
• Test Pilot School graduate or Weapons Instructor Course graduate with more than five years of test experience.
• Minimum 1000 hours of flying time.
• Active TS/SCI clearance.
• Staff/Program Manager experience
• Master’s degree in engineering or related technical field.
• Experience with/knowledge of development and testing of electronic warfare mission data and the electronic warfare reprogramming processes.

To apply for this opportunity or see what other open opportunities are available at Bevilacqua Research Corporation, please visit our careers page at https://portal.brc2.com/careers/.

Bevilacqua Research Corporation. is an equal opportunity employer Minorities/Females/Disabled/Veterans and VEVRAA federal contractor. We recruit, employ, train, compensate, and promote without regard to race, color, religion, sex, national origin or any other basis protected by applicable federal, state or local law.