Testing methodology

Checkvir Virus Testing Laboratory of Veszprog Ltd. announces its realtime antivirus testing service.

This service is based on an automatic system that is able to check almost all released versions of antivirus solutions.

Note: The system is able to detect if an antivirus is updated. This detection takes a while which is less than 30 minutes. If there is more than one new version during this period then only one of them is detected as a new version, others will be ignored.

Testing procedures are executed automatically using a special frame system. This automatic system provides a database accessible on the Internet including the test results related to each version. This system includes the following parts:

Clients: These computers are able to execute different tasks, such as updating or testing. Client computers have exactly the same hardware and software. (Even the cards are inserted into same slots.)

Update task:. A debian Linux system and perl scripts are dealing with changing the image of the Windows operating system (including the antivirus) and execute them periodically. The Windows system includes installed WinTask software and an updater script written for this software, which tries to update the antivirus. If there was an update then it will be indicated to the controller under Linux, which will create an image including the new update to a network storage.

Testing task:. For security reasons computer executing testing tasks are connected to the internet via a special firewall. A debian Linux system and perl scripts are dealing with changing the image of the Windows operating system (including the antivirus) and execute them periodically. It loads the images saved by the Updaters. Depending on the test procedure(s) it may copy different Wintask scripts into the image that will be executed after the Windows operating system has been booted. Perl scripts under debian Linux have to prepare the data used for testing (e.g.: samples, clean files, …) and they have to save the results as well. In the case of new results they are analysed and they are written directly to the SQL database of the Webserver computer.

Webserver: It collects test results in its SQL database and provides it accessible via its web page.

Archiver: All of information about executed tests is archived by this computer. It includes test results, log files and images as well as data required for testing.

Controller: This computer manages the whole process on different parts of the system.

Firewall: There is a firewall between the inner (red) and outer (blue) networks. It is used for managing the system only, it is not required by the automatic working mechanism of the system.

Firewall & router: There is a special firewall and router among the client computers, the "malware proxy" server and the internet. The main task of this computer is to distinguish between the network traffic of the virus protection and the malware. The traffic of the malware is forwarded to the "malware proxy" server however the network traffic of the virus protection is forwarded to the internet. Thus solutions may use internet connection, so the usage of "cloud technology" is not limited.

"Malware proxy" server: This server can store the content of malicious sites related to the certain time. So it can simulate the tested part of the internet for clients. It is used in the case of dynamic testing (when the malware code is executed, or a malicious site is opened).

Testing procedures are related to the user requirements. Antivirus solutions are regarded as black boxes. The following testing procedures are executed:

Malware knowledge (detection): Tested solutions have to be able to generate a report file about all detections.

Solutions are executed on the set of malware samples. Analyzing report files detections are counted by each malware and they are classified into three categories: malware detection, suspicious code detection and detection of a non-malware code (e.g.: packers). This classification is based on the information provided by the particular vendor.

Malware knowledge (disinfection): Tested solutions have to be able to generate a report file about all actions (disinfections, deleting, renaming, ...). Solutions are executed on the set of malware samples. Analyzing report files and the disinfected storage disinfections are counted by each malware.

Speed test (in clean environment): Solutions are executed on the set of clean samples while automatic actions in case of detection are set to do nothing. Required time for scanning is measured.

Container test: Solutions are executed on the set of containers including a known malware or the EICAR.COM file. Container means every files that can contain other files inside, such as packers and email client softwares. The main purpose of this testing procedure is to check ifthe user would like to test the whole system then the protection is able or not able to detect malware in containers inside. For comparative reason the same known virus sample or the EICAR.COM file is used in the case of all antivirus solutions. This test procedure include the following tests:

• The scanning capability inside different types of containers.
• The scanning depth in the case of joined containers into each other.
• The scanning capability in special cases: using long file names, password protection, ...

Testing procedures can be static or dynamic. In static case malware code is not executed, however in dynamic case malware code is executed while the behaviour of the protection is monitored. In the case of dynamic testing and in every speed test always native environment is used. Virtual environment is used only in static testing if it is not related to the speed.

Static testing procedures are executed in on-demand and in on-access case. On-access test is executed using the following command line command:

copy <testsample> <target directory>

In addition the test results of knowledge tests, false positive test and container test are compared. In case of speed tests the procedure is executed in the same environment without antivirus and the time values required in this case are subtracted from the on-access speed test results. All of test procedures are related to speed test are executed minimum 30 times and the minimum, maximum and average time values are calculated.

Dynamic tests are executed in native environment. During these tests malware codes are executed or malicious URL-s are opened. In the testing system all of requests to the internet caused by the malware are forwarded to the special "malware proxy" server. It means that all of testing case according to different solutions can receive exactly the same bitraw from the internet.

The following test sets are used for testing:

Malware test set:

Malware samples are malicious files and malicious URL-s as well. Malicious files are collected using the honeypots of CheckVir Lab, Offensive Computing and vendors. Malicious URL-s are continuosly archived by the "malware proxy" server using database of accessible black lists on the internet. All of used samples are previously verified for proofing that they are really malicious and working as well. After the verification of samples an in-the-wild set of them is selected for testing. For this purpose prevelances related to malware published by vendors, by other companies or organizations are used, such as IEEE ICSG malware working group.

Clean test set: This set includes non-malware files of different operating systems and applications AND separately an amount of packed files using different exepackers. Thus exepackers blacklistings can be reported.

Container set: On the first hand this set includes data files of the following containers:

• Storage files of email clients: Microsoft Outlook, Outlook Express, The Bat!, Eudora, Pegasus Mail, Mozilla Thunderbird, Incredimail
• Data files of packers: ZIP, ARJ, RAR, GZ, TAR, TGZ, ACE, CAB, JAR, LZH, BZ2, HA, Z, 7Z, BFC
• Embedded files: DOC, XLS

On the other hand original malware samples are included in this set as well as other specially constructed

container data files according to the mentioned special cases in the related testing procedure.

Solutions are provided by the vendor. A solution must not be only one software, it may include more software

as well. Usually vendors are asked for the best suggested solution.

They must submit the following additional information also:

• Vendors have to specify the suggested settings for home users and for business users as well. (It is usually the default settings.) All of testing procedures are tested using these two settings. Vendors may submit different solutions for home and business users.
• Vendors have to specify the classification instructions related to distinguish among malware, suspicious and informative detections.

Vendors have to submit the solution and specify the suggested settings that they have to perform the following requirements:

• Solutions using the given settings have to be able to create one or more report files about all detections and disinfections.
• Automatic actions have to be performed in the case of detection.

Note that all test cases are executed using the same settings.