comparative analysis on e-gov web sites

Web sites usability comparative analysis

HumanComputer Interaction

A project about web applications’ interfaces usability in specific business contextes (Web 2.0)

Rev 1.5All Rights Reserved - ©2009/2010

Started on 11/25/2009, Released on 09/12/2009

HCI ACADEMIC COURSE - TUTOR: PROFESSOR D. GIORDANO

www.defense.govwww.difesa.it

Andrea Tino - University of Catania - Research Paper - Human Computer Interaction - AY: 2009/2010

Processing military/defense industry:US Department of Defense vs Ministero della Difesa IT

Where and how we’ll moveWhen analyzing web sites, especially when comparing more than one web application, it is necessary to focus on industry. We are taking into consideration two organizations belonging to the country military defense industry (for this reason they are competitors); this particular context has special characteristics, and web sites representing an Internet access point to such institutions, must meet some requirements. Detecting these requirements makes easy to evaluate our web sites, if all conditions are met, then the web applications we are analyzing provide a good service.

What services web sites representing country defense institutions should provide

Getting into the clothes of the user who is navigating through the web application in exam, the first question we must answer to is: “What services do I expect to find?“. Another useful question is: “What kind of users would access the web site?“. Although these two questions may appear as useless as point-less, they help us to find out our requirements for the web sites we are going to examinate. So, let us answer the questions.

User typologies

Web applications like the ones we are considering, are visited by internal personnel (for accessing private areas, private services and so on), by newsmen (to get news and press releases by voicemen and official sources), by citizens and users involved in the military and institutional fields (like politicians, military personnel and so on). The most important point we must reach, is that the web sites in exam are con-sulted by a very wide range of people belonging to various divisions:

• Politicians

• Military men

• Civilians

• Military or civil institutions

• Privates

Purposes and communication objectives: providing services (analysis on usability)

All these typologies individuate functionalities that the web application should provide; in fact the main purpose of both web sites is to providing a set of services accessible to users. But we are not interested in the services’ content (for the moment); we want to analyze how the system provides access to these elements and how it makes them usable for the user. For this reason we can answer to the other question and detect the main services the two web sites should afford:

Context at a glance: Country Military/Defense industry


Pg. 2

• A constantly updated news area where it is possible to get information about the most important is-sues regarding country defense system and country threat present level.

• A specific area reserved for civilian and military leaders and heads of the country. This service should provide access to the main hierarchy in order to detect immediatly people involved in the various fields of the country defense departments.

• A dedicated area for past, present and future missions in homeland and foreign territories. This area is supposed to provide information about soldiers dispatch and mission objectives and status.

• An area providing access to the main alliances between the country government and other govern-ments in the world. This area should also provide information regarding international alliances (i.e. NATO and similar).

• An area providing links to the main defense branches. This list is supposed to provide a connection to the main military and non-military institutions involved in the country defense system just like Army, Navy, Country Defense Headquarters, Auronautics and so on.

• An area dedicated for users who wants to join the defense system as member of the civilian person-nel or the military personnel. This service should provide links to the main institutions ruling the recruitment/employment procedures.

• A connection to the main judicial branches of the country. This connection is supposed to provide information about the ordinary laws and the military laws regarding the most important national and international issues about war and conflicts.

This collection is a first draft about the main services that both web applications should provide to the user (and will be revised later in the next chapters). When analyzing the two web sites, our work is simplified because we can focus on the services and then make comparisons.

Usability criteria are many and it is easy to get lost among them, focusing by services is a way for setting a pattern in our analysis, the list above will be a sort of guideline in our evaluation process.

Introduction to difesa.it and defense.gov + initial con-siderarionsBoth web sites act in the military/institutional fields and they provide a connection to the country de-fense and homeland security departments. For this reason they are not just simple web applications navi-gated and consulted by users; they provide services and information which are useful for a wide range of kinds of people: civilians, military men, politicians and so on.

At a first glance they look very different in setting and in their graphic structure (we are not begginning our analysis, we are just having a first look to the applications, without going deeper), but their differ-ences rely in other aspects; for example, given the fact that they work for homeland security, both sites use different main languages: American and Italian (we still don’t know if one of them also provides sup-


Pg. 3

port for different cultures). They also refer to different hierarchical systems: in Italy the homeland security department is represented by the Ministero della Difesa, while for the U.S.A. we have the Department of Defense, and it is important to underline that these two institutions have different rules and a different internal organization.

All the differences don’t really rely on the website itself, it is a matter of institutional correlation. The homeland defense department has a deep connection with the country political organization; for this reason, services provided by the two web applications will be various and sometimes they won’t have any meeting point. This fact will complicate our path based on services description (we have to detect common services and try to generalize those services which both applications provide in a different way), but, once finished, our analysis will not be affected by any external problem.

Websites usability contest now begins: US Defense Department vs IT Ministero della Difesa

Now we can officially start our analysis by introducing the main actors: the websites we are going to compare.

Ministero della Difesa della Repubblica Italiana: www.difesa.it

United States Department of Defense: www.defense.gov


Pg. 4

A system for assigning scoreThrough our analysis we have to take into consideration various characteristics for both the sites. In the next chapters we are going to explain the path to follow for covering all elements inside the applications (in order not to miss anything), but before doing that, it is necessary to specify which methodology we are going to use for assigning a “score“ to every evaluated element.

Element: definition

Although we still have to define what we are going to evaluate (elements), we need an initial definition of “element to evaluate”. With the expression “element to evaluate“ or simply “element“ we mean any application’s characteristic which it is possible to assign a score to, indicating the application’s attitude in implementing that aspect (the higher is the score, the higher is the characteristic’s quality).

Element importance

We are going to examinate many elements, but not all of them have the same importance in the site evaluation; meaning that although two elements are evaluated with the same “score“, they may not as-sume the same value inside the global evaluation. In order to take into consideration this situation, we need to tile another evaluative quantity (a second score) to our present score: this entity is called ele-ment’s weight or element’s importance and it is a numerical quantity assuming these values:

• Irrelevant - 0.2: The considered element has not a sufficient importance in the global evaluation process; if present, the element is just an available feature.

• Considerable - 0.5: The considered element is not needed in the global evaluation process.

• Important - 0.75: The considered element is important in the global evaluation process.

• Determinant - 1.0: The considered element is really important (necessary) in the global evaluation process.

Element score

We can now define the score assigned to an element. We define score as a numerical quantity indicating the quality of the element’s implementation proposed by the application:

• Very bad - 0.0: The considered element’s implementation is absolutely not good, doesn’t work as expected at all, and doesn’t provide a useful service.

• Not good - 1.0: The considered element’s implementation provides a poor service not satisfying all requirements and not working as expected (in some aspects).

Evaluation criteria and examination path


Pg. 5

• Good - 2.0: The considered element’s implementation provides a good service satisfying many requirements and working as expected (in almost all aspects); this level defines a minimal quality threshold.

• Very good - 3.0: The considered element’s implementation provides a reliable service satisfying all requirements and working as expected (in all aspects), this level defines the best quality threshold.

• Excellent - 4.0: The considered element’s implementation provides a reliable and robust service satis-fying all requirements and providing access to other useful functionalities (even if not required); the element works as expected (in every scenario).

Six evaluation dimensions

An element always represents an aspect of the application, we organize all aspects and characteristics into six groups (dimensions) so that, finally, we can represent the global evaluation process result into one single radar chart. These dimensions are:

• Architecture: Site structure and navigation system.

• Communication: Attainment of the communication objectives.

• Functionalities: Correctness of the site functionalities.

• Content: Quality of the informative content (information and data).

• Management: Site operativity (site works fine?).

• Accessibility: Site accessibility (is site accessible to all kinds of users?).

Totalizing dimensional score

We are going to group elements by dimension and assign to each one a score (concordently with the assumpions made before), but in order to get a final score for each dimension, we need to find a math-ematical procedure.

• Let D be the dimensional score (score referred to a particular dimension).

• Let n be the elements in the dimension.

• Let Pi be the element’s importance (weight).

• Let Ei be the element’s score.

Given these hypothesis we consider the following formula:

It formerly represents a weighted media where weights are element’s importances and scores are insert-


Pg. 6

11

1 1

1

n

i i n ni

i i ini i

ii

P ED P P E

P

−

=

= =

=

⋅ = = ⋅ ⋅

∑∑ ∑

∑

ed inside the mathematical process. For every dimension we get a dimensional score to use for creating the final radar chart.

Totalizing site evaluation score

We also provide a numerical quantity useful for resuming all aspects of the site, without considering dimensions. This number can be obtained in this way.

• Let S be the final site score.

• Let n be all the examinated elements.

• Let Pi be the element’s importance (weight).

• Let Ei be the element’s score.

Given these hypothesis we consider the following formula:

Doing so, we will be able to absolutely determine which site is better than the other just by making a numerical comparison using this quantity.

Our next stepsOnce the scoring methodologies and formulas are defined, we are now able to begin our analysis. Focus-ing on dimensions (in the order shown before), we are going to define elements to work on. For every element we are going to assign a weight and examinate how both sites implement it, in order to assign a score to each web site. Once all elements are finished, we finally get the dimensional score for every site. This procedure will be performed for every dimension.

Now we can start examination process and evaluation.


Pg. 7

11

1 1

1

n

i i n ni

i i ini i

ii

P ES P P E

P

−

=

= =

=

⋅ = = ⋅ ⋅

∑∑ ∑

∑

Site architecture dimension defines all those elements regarding the web application’s page relations, links structure, user navigation system and site map. Basically we are now going to examinate how the web application organizes its content and provides information to users.

Web site structureUsing the expression “site structure” we mean the site’s pages organization. Every web site uses its own pages structure so that it is possible to indentify many known patterns, recognizing the site structure let us refer it to a known pattern, and every known pattern has known defects and known esteems.

Element importance

The structure of a web site is crucial for a good usability level, the better is the structure the more con-fident will be the user in navigating the web application and the better will be his use experience. We assign to this element importance Determinant (1.0).

Getting site structure

To get the structure of a web application it is necessary to navigate the web site through its pages. Our web sites have very different internal organizations, this makes different the navigation flow too (naviga-tion flow will be described later). Starting from the home page and getting deeper in the structure we can draw a simple diagram, we should not be (in this place) too much detailed, so it is possible to omit secondary pages and focus on main areas.

Examinating structures

We can see below both web sites’ structure, it is possible to notice many differences relying in general setting and pages disposition in the diagram. Beginning our analysis we can focus on setting. In the US Defense site we can notice a hierarchical order starting from the home page, going to the six main areas (News, Press resources until Contact us) and ending up in the second level pages (each one belonging to a specific area); basically the entire structure is divided in two levels, the secondary level contains detailed elements shown in the first level. The IT Defense site uses a different approach: all pages are, in fact, inserted in a common level so that it is impossible to identify a hierachical scheme.

Comparing both diagrams we can find usability goals and failures. At a first look at the IT Defense site’s structure, while its monolithic disposition may appear as dispersive and chaotic, it provides a single access point to all application’s areas; this approach must not be underenstimated: in fact, sometimes (es-pecially for this kind of web sites), users need a fast gateway for entering requested pages. The completely different US Defense site’s structure defines, instead, a good organized disposition of pages that are reachable by the user each time he enters a particular area; this approach compels the user to navigate other pages in order to access the desired content (because of the hierarchy); although

Evaluation for Architecture


Pg. 8

this organization implies more effort by the user, it gives a good logical map to him; in fact, if on one side we have this usability miss (apparently), it is demostrated (on the other hand) that users are able to perceive page paths and patterns (if pages structure is very good) and locate autonomously the desired page (even if it is the first time they navigate the web application).

Element score

Because of all considerations made before, we find a condition of parity in both diagrams, the score will be the same for the web sites. However, notwithstanding the positive considerations made, we won’t as-sign the max score because both sites are not absolutely simple to navigate: the US Defense site is cha-otic in some pages, generating many loops inside the diagram we didn’t show for space needs, and the IT Defense site uses different pages to organize correlate content (leading user to confusion).

SCORE FOR US DEFENSE SITE IS: VERY GOOD (3.0)

SCORE FOR IT DEFENSE SITE IS: VERY GOOD (3.0)


Pg. 9

Web site structure for defense.gov

Web site structure for difesa.it

Web site navigation (tools and menues)While a standalone client application is generally utilized by the user, a web application is navigated by him. This concept is important to understand because it focuses on a truth: “a web site is a complex col-lection of views through which the user can access the provided functionalities and content defining, so, several different browsing flows“.

Element importance

Site navigation is an expression indicating a web application’s ability to define a set of tools able to lead the user through the site’s structure. We considered, a while ago, the structures of both web sites; site navigation is a correlate element because a good structure is nothing without a good navigation system. Navigation is in charge of defining the best patterns the user will follow to access the site’s pages. Because of all these considerations, this element has a high importance level: Determinant (1.0).

Navigation tools

A web site generally implements a set of tools in order to help the user to navigate the site structure. One of this features is of course the site map, but this is an element we’ll see separately (given its importance). There are other controls provided to users for site’s pages browsing purposes, for esample menues and navigation bars. Both web sites uses menues but with a different approach.

The US Defense site defines a top menu showing the six main areas located in the primary level of its site structure (see the diagram in the previos paragraph); when entering a secondary level page (depending on which first level page the user comes from) top menu may change (in graphic and content) showing more options for the selected area, or stay the same. The site also offers another menu localized in the bottom part of some secondary level pages (especially in pages examinating top leaders and American defense department personnel), it shows external links (always regarding the American defense institutions) or more options for the current page, sometimes it also duplicates the links shown in the top menu of the same page. The US Defense site finally uses a side menu showing inner levels’ options. In a lot of pages, the side menu organizaes a set of links (by argument) in a very good list structure, the user is able to select pages regarding the same argument of the current page without losing its position, because the side menu is always available in pages shown inside it.

The IT Defense site, given its one level structure (see previous paragraph), uses one side menu showing the unsorted list of all areas in the site structure. In detail pages the web site offers also a second side menu (on the right) showing links regarding the current page and sometimes having the same title of links in the left menu, these links are not the same because they lead to different places; all links shown in the right menu lead to indicated pages focusing on details correlate to the current page. There is, finally, another navigation tool provided in many pages of the site: a site map path located in the top area of the page showing the current user location in the sites structure; not all pages implement this feature.

Examinating navigation

The US Defense web site, with its navigation features, provides to the user various tools for jumping to desired pages; in this way the user is always surrounded with many links (in the top, side and bottom


Pg. 10

areas) and has a high interaction level with the application. We must also consider that it is the typical American surrounding strategy the one used in this place; in fact, placing banners, links, animated gifs and interactive tools all around the page’s main content, it is possible to create a perimetral interaction region where user finds all the links he needs; Americans are accustomed to this template and for this reason they are very familiar to it. That’s why we evaluate the US Defense navigation approach as very good because it matches the user needs (American defense web site offers information mainly for Ameri-can people).

The IT Defense site doesn’t offer a very useful navigation structure, because the main side menu (the left one showing all areas) is chaotic and unsorted, there isn’t even a grouped scheme, links are all presented as a list. It is demostrated that long lists are very tedious for the user to utilize because it is difficult to search for a specific link. Furthermore, the right side menu showing page context based links, is not so immediate to use; the user is confused at the beginning, seeing two similar menues, and there is not a description explaining the purpose of both navigation lists. All considerations made up until now lead to an usability failure: in order to avoid confusion and link-ing chaos, it is necessary to use one menu only and using, for inner pages, a tree structure; here we see a navigation based on two unsorted lists, the user is really compelled to watch both sides of the page in order to detect what he needs and this is a very difficult situation for him. A positive factor is represented by the horizontal top site map path; users feel much comfort in seeing such a navigation tool because it is like a map with a large red point with the text “You are here“; this is sure an usability goal, even if we must take into consideration that not all pages implement this feature.

Element score

Given all considerations made before we can extract the final score for both web applications. US De-fense site has a good navigation structure, user may encounter an initial difficulty in seeing so many links, but we must keep in mind that this is an American commonly used graphic pattern that renders this visualization a good way for navigating the web site. IT Defense site has many usability misses that are not covered by the horizontal map path navigator (which is just single white stone in a box of black ones).


SCORE FOR IT DEFENSE SITE IS: NOT GOOD (1.0)

Site mapIn a web application the site map is a good feature that enables the user to look at our site just like if he were to watch a city from an observatory. This is not a required feature, but sure is something simplifying user navigation. We are treating site map as a separate element from navigation element, because we think of it as an external tool not so strictly connected to site navigation (even if some connection points are present).

Element importance

We already said that site map is not a required tool, this implies that importance level, for this element, is lower than Determinant.


Pg. 11

In this place we consider site map as a useful tool, if a site implements it then usability level grows. Be-cause of this reason we mark this element as Important (0.75).

Maps analysis

Both sites implement a site map but using completely different approaches.

The US Defense site has a very common graphic pattern used for site maps: it shows a grouped list of links to the main areas of the site’s structure in the absolute bottom part of the page. When the user ac-cesses the home page, after scrolling to the bottom region, he can see a collection of lists organized in groups and divided in different columns (five columns). Links are grouped by area and, globally, it is pos-sible to recognize a structure similar to a distributed tree. Even in this details we notice the presence of a common American graphic pattern, so we can also imply that American people are accustomed to this kind of maps.

The IT Defense site uses, insted, a common Italian graphic pattern for web application maps: that is an entire page (reachable by a link located in the bottom part of every site’s page) dedicated to the map and showing a list of the main pages grouped by areas.

Examinating maps

The US Defense site map is simple and has a very special particularity: depending on which page the user is navigating, the map adapts itself showing more or less links in order not to show useless information (just like uncorrelate pages or redountant links). By doing so, the user has always a sort of pocket guide available, where all directions are showed depending on his position.

The IT Defense site has, instead, a very common structure regarding its map; that’s why we don’t go deeper in our analysis being unable to describe special features or similar.

Element score

It is evident that US Defense site reaches a very good usability level (considering also that all its map is viewable in a small part of the page while in IT Defense site the map uses all page’s space and user needs to scroll). Regarding IT Defense site, we have just a simple service not showing particular features. Given all considerations made up until now, we express these final scores.

SCORE FOR US DEFENSE SITE IS: EXCELLENT (4.0)

SCORE FOR IT DEFENSE SITE IS: GOOD (2.0)

External links and communityAlthough it might seem that external links are not part of a web site’s architecture, they provide a service, for the user, to reach other web applications that are probably connected to the main one (for various reasons). That’s why, in our evaluation process, we will consider external links as an element to assign a score to.

Brief considerations on external links and community connections in web sites rep-resenting institutions


Pg. 12

It is a common mistake considering that external links have the same value in all web applications. It is not the truth because the relevance of external links is deeply connected with the web site’s context/industry. In our case the institutional field is a very particular context. When a user is navigating the web site of an institution, he expects to be able to access other institutions in order to get other services, or also expects to find a connection to the seervices in some community engines. Today communities are a widely diffused concept, almost everyone has an account with Facebook or Twitter and so on... An insti-tution having also some instances of its services in a community web site surely is a very good service provider.

Element importance

Considering what we said just two lines before, we consider this element to be important in a web site, for this reason the importance level is set to: Important (0.75).

Analyzing external links structure

Both US Defense web site and IT Defense web site have external links. But even in this case it is possible to notice many differences.

The US Defense site places community links (and a few link to other military institutions) in the home page and in the contacts section on the left side of the screen. We can see a wide range of external areas connected to the US Department of Defense just like: DoLive, Facebook, Flickr, Twitter and YouTube. There are also links for subscribing to the defense.gov site newsletter: RSS Feeds, Podcasts and Widgets are just an example. In the DoD Websites area of the web site, the user can find all external links he needs thanks to a very useful page showing all links in an alphabetically sorted table (where the user can choose a letter in order to show all links having the first letter corresponding to the selected one).

The IT Defense site uses a different approach. As for the site map, links page has a completely dedicated page. And also in this case, all links are presented in a list. This time the list is sorted by area generating a structure similar to a tree. In this case, the web site shows the list of institutional links only, there are no links to community engines.

We can say that US Defense site has a high usability level because all links are viewable without scrolling the page and have icons next to them that helps the user to find out what he needs, even without read-ing what’s written. This feature (readibility) is present also in the page for external links where navigation by letter avoids long and endless lists very difficult to consult. On the other hand, we also need to consider that IT Defense site makes all mistakes described before.

Element score

Given all considerations upon both sites’ external links, we set these scores:



Getting dimensional score


Pg. 13

We can now get the final score for this dimension using the formula introduced in the first chapter.

As we can see US Defense site has a higher usability level regarding the Architecture dimension; the dif-ference with the IT Defense site is considerable.

ARCHITECTURE DIMENSIONAL SCORE FOR US DEFENSE SITE IS: 3.40

ARCHITECTURE DIMENSIONAL SCORE FOR IT DEFENSE SITE IS: 2.00

In order to better see the situation, we can chart data and results evaluated up until now in a bar diagram showing both scores and the mean between them.


Pg. 14

( ) ( )( )

1 14 4

1 1 1 1

11,0 1,0 0,75 0,75 1,0 3,0 1,0 3,0 0,75 4,0 0,75 4,03,0 3,0 2,96 2,96 11,92 3,40

3,5 3,5

n n

i i i i i ii i i i

D P P E P P E− −

= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + + ⋅ ⋅ + ⋅ + ⋅ + ⋅ =

+ + += = =

∑ ∑ ∑ ∑

( ) ( )

1 14 4

1 1 1 1

11,0 1,0 0,75 0,75 1,0 3,0 1,0 1,0 0,75 2,0 0,75 2,03,0 1,0 1,5 1,5 7 2,0

3,5 3,5

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + + ⋅ ⋅ + ⋅ + ⋅ + ⋅ =+ + +

= = =

∑ ∑ ∑ ∑

Charting score for this dimension

0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5

Mean

IT Defense

US Defense

Every web site defines a communication level where the main objective is fulfilling the application pur-poses. In our case, we have already defined both sites’ purposes and communication objectives in the first chapter; so we can start analyzing those elements connected to the communication level.

Home pageHome page is the first thing a user looks at entering a web site, it is calculated that the medium user ac-cess and first navigation time for a web application consists in few seconds (10, 20 and in some cases also more than 45 seconds) before he leaves. Home page, has for this reason, a very important target: com-municating the largest possible quantity of information to the user, in order to allow him understanding what the web application is about.

Element importance

Web sites’ home page importance is recognized by all users in the world and all web developers and programmers; it is useless to spend time for explaining why we give the highest importance level to this element. Importance: Determinant (1.0).

Home page analysis

We are going to proceed by questions. The first one is: “Does the home page communicate directly and in the fewest time possible, what the entire web site is about?“ and the others are: “Does the home page use any payoff or tagline?“ and: “Does the home page provide any description or any special content to inform the user about site’s details or information?“.

The US Defense site’s home page has several characteristics. It is 1000 pixels wide and screen centered, all content is easily viewable without scrolling the page (the hidden part includes the site map only); there is a top image title, the main navigation menu, a side menu for quick external links and a centrered news spot area with animated content changing every time. The American circular symbol with the eagle (in particular, the American eagle holding three arrows refers to the American Defense) on the top area, helps the (American) user to recognize that the site he is browsing is an institutional oriented application. The background color also underlines the institutional aspect of the web site (blue is widely used in the U.S.A. for pointing out internal institutions, this is an example of differences between cultures, in Italy this aspect connected to color is not as strong as in the U.S.A.). American defense home page doesn’t use payoffs or taglines, there is just one significant text: “United States Department of Defense“ giving the whole idea about the site context. There are also no descrip-tions or information about the web site. In the end, images and animations showing soldiers, marines and military personnel at work (with a tex-tual description for every item), renders impossible to misunderstand what the site is about.

Evaluation for Communication


Pg. 15

The IT Defense site’s home page has different characteristics. It is 750 pixels wide and screen centered; the graphic structure defines a top image, a top menu bar and the main body divided in three columns showing a side menu, a central content with a news image (static) and a banner/news side area. The top icon showing the Italian Republic symbol and the Italian Flag’s colors give the user a good start-ing point in order to understand what the web application is about. The user would probably have no doubt about the site’s content after seeing, on the other side of the page on the top area, the flag of the Ministry of Defense showing the four military forces and a central star. Going down from the top region, the page is divided in three vertical parts. The user must scroll the page in order to see all content, and the page is very long, too much long for a home page. The news image in the central part of the page changes depending on the headlines but there are no descriptions apart from the news title.

Performing a last examination about links provided and site’s areas access controls, we can see the US Defense site shows a lot of links and connections to other related institutions, and all links are available without scrolling the page with significant icons and images. From this point point of view, IT Defense site does not provide the same usability level: there are some links refering to other Italian institutions but they are reachable by scrolling the page only.

Element score

Given the results of our analysis, we can see that the American defense web site has a very good home page, showing all information in the most reasonable way, and allowing the user to understand in few seconds the site’s purposes and communication objectives. A different result comes from Italian defense web site: the home page sure is able to communicate the web application’s purposes but it is not as immediate to understand as the competitor’s home page does; this is probably an usability failure caused by the poor graphic structure, especially regarding the total page height: the home page is not enough wide and is too much long. There are also too many in-formation that are not presented in a suitable content structure, in fact there are too many lists and even they are too much long.



Brand imageBrand image is a particular feature provided by home pages, or other pages, in order to let the user un-derstand what the web application is about in the fewest time possible.

Element importance

Given the sites context, brand image is not a required element, brand images are useful elements for web sites selling products or similar. For this reason we give a low importance level to this element. Impor-tance: Irrilevant (0.2).

Brand image analysis

Both sites don’t really implement a brand image, the only one image indetifiable as a brand is the news image. So we cannot say much about this element.


Pg. 16

In particular, the US Defense’s home page has a good brand because news image changes everytime showing significant pictures concerning the military/defense site context. IT Defense’s home page has just a static image; pictures shown, depending on headlines, concerns often political events, for this reason the brand level is low.

In other pages, the US Defense site shows often a wide picture related to the area the user is navigating. The IT Defense site, instead, doesn’t use brands in pages different by the home.

Element score

Given the considerations made up until now, we can set a score for the brand image element.



Graphic structureIn the programmers/web developers & designers world there is a still open debate regarding graphic vs textual content and which element is more important than the other. In this contest the question is: “What’s the most important part in a web site to focus attention on: information, data and textual content or graphics?“. As we said, the debate is still open, but there are few good answers; one of this explains how, from usability point of view, the graphic structure of a web site is the only one element providing the user information on what the web site is about without any need of reading any line of text. For this reason graphics is important. In this paragraph we’ll analyze this element for both sites.

Element importance

Considering what we said few lines before, there is no need to explain why we choose the highest impor-tance level for this element. Importance: Determinant (1.0).

Analysis on page layout

Every web site defines two sets of pages different from the graphic structure point of view: the home page and all the other pages. So, apart from the home, all pages of a web site have the same graphic layout.

The US Defense pages’ layout has the following characteristics:

• Pages are width fixed and all 1000 pixels wide. They are browser centered.

• Under the top image, there is the main menu with textual links. The main body is divided in three sec-tions: a side submenu, the content area and the features/headlines panels. Almost all pages have this structure so that the user acquires familiarity with links and controls disposition. In the bottom region there is the area map, showing pages related to the current one.

• The content area is always filled with text and images; text has always the same structure with boxes showing related data.

This is undoubtly a good page layout even if, given the large quantity of information, it would be better


Pg. 17

using pages that covers all browser width.

The IT Defense pages’ layout has the following characteristics:

• Pages are width fixed and all 750 pixels wide. They are browser centered.

• Pages are divided in four regions: a top region with page title and the main image showing the of-ficial logo of the Italian Ministry of Defense, and the main region divided in three parts: a side menu, a central content area and anothe side submenu.

Even in this case we think that page layout width is not suitable for this site, but in this case we don’t sug-gest to adapt the width to the browser area, it is sufficient to change the size of the page to 950 or 1000 pixels.

Focusing on color schemes we find many differences between the web sites. The American Defense site uses brilliant colors; the chromatic range varies from blue to white, but always inside the cold frequencies (blue, light blue, dark blue, white). Text uses other colors: red, orange, black and white with blue background (in case of links some colors changes). Italian Defense site has a different color range, in particular the main difference with the competitor is that this site uses light colors: light grey, light yellow, light blue and sometimes light violet. This scheme is not recommended by web design guidelines, because the absence of strong gradients renders difficult the user navigation.

Analyzing gestalt, both sites reach a good level but, even in this case, the American Defense site has a better approach than Italian one. The reason relies in colors: in the U.S.A. we have already said that blue is a color associated to the institutional field, American culture renders easy to identify web sites working for/as institutions while in Italy this situation is not present.

There are still few aspects to analyze bofore setting a score, one of this is typography. The US Defense site uses common font families for text (compatible with many operating systems). There is a wide use of character styles just like bold, underline, background images for titles, side images and colors. Clickable and dynamic text has a different style in order to be recognized by the user (especcially for news abstacts), although there are various link styles, all links are underlined or show a little arrow at the end of the text. In IT Defense site, text has few styles in order to differ a paragraph title from the body. Links are recogniz-able because of the underlined style applied to it, but sometimes clickable text uses other graphic tap-perns (some links are underlined while others are bold or others are also boxed), this is not a good thing because some links are not recognized by the user.

Focusing, in the end, on attractiveness, we find two different worlds. Although this kind of aspects is too much subjective, a good way for evaluating this parameter is focusing on images and styles. US Defense web site, from this point of view, is the more attractive one, because of the number of textual styles it uses, the number and quality of images and the site color scheme; Italian Defense site has a bad color scheme and there are not many textual styles and icons. If we were to set a judge about attractiveness, we would probably say that US Defense reaches a very good level while Italian Defense gets a sufficient level.

Element score


Pg. 18

There is not much to add to what we have said up until now, we can set the score.



Getting dimensional scoreUsing the same procedure of the past chapter we can get the final dimensional score. As we can see US

The US Defense site has a higher usability level regarding the Communication dimension; the difference with the IT Defense site is considerable.



In order to better see the situation, we can chart data and results evaluated up until now in a bar diagram showing both scores and the mean between them.


Pg. 19

( ) ( )

1 13 3

1 1 1 1

11,0 0,2 1,0 1,0 4,0 0,2 4,0 1,0 4,04,0 0,8 4,0 8,8 4,0

2,2 2,2

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + ⋅ ⋅ + ⋅ + ⋅ =+ +

= = =

∑ ∑ ∑ ∑

( ) ( )

1 13 3

1 1 1 1

11,0 0,2 1,0 1,0 2,0 0,2 2,0 1,0 2,02,0 0,4 2,0 4,4 2,0

2,2 2,2

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + ⋅ ⋅ + ⋅ + ⋅ =+ +

= = =

∑ ∑ ∑ ∑


0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0

Mean

IT Defense

US Defense

Services, functionalities and interactivity provided by web sites must satisfy user needs and must also be easy to understand and to use. We are now approaching to study these aspects of both web applications evaluating what functionalities they provide (apart from search function that will be described in a dedi-cated section) and then examinating their quality from several points of view.

Implemented functionalitiesLet us first detect what functionalities both sites provide to the user. Listing them, we’ll be able to make comparisons and see which site has more services.

Element importance

User expects to have some services in the web site he is navigating, functions and services are essential because they are the core of the application. That’s why we give the highest importance level to this ele-ment: Determinant (1.0).

US Defense services

The American Department of Defense web site has a lot of functionalities that improve user interactivity with the system, here we show those ones we believe to be the most important and useful:

• Threat advisory: A tools always available on the top right side of the page informing how much high is the present American military and homeland defense emergency level. This functionality is very useful and provides a constantly updated data about homeland security.

• External link disclaimer: This function enables the user to interact with the web site, allowing him to notify links that does not concern the American defense web site mission, in order to improve the web application infrastructure. The user is requested to write a description about those links he believes being inappropriate, and he is also requested to submit those links he believes ought to be included in the web application. Sure this is a good service because, doing so, the system traces user prefer-ences and it is also a good way for improving usability.

• Interactive survey: This is a tool available only for those users navigating many pages of the web site. When the application detects a user who never entered the site before, a request for submitting a survey about the American defense web site appears. Users can reject or accept, in case they accept, the application shows them

Evaluation for Functionality


Pg. 20

defense.gov survey

a series of questions (in a new window) the users can answer to. At the end, a message tells the user that his choices will be considered in order to improve the web site services for matching his prefer-ences, among with all the other users’ ones. This is a good functionality for the same reasons seen in the previous point.

• RSS, Podcasts and e-mail subscriptions: The web site offers links for allowing the user to subscribe to e-mail alerts and newsletter; there is also the possibility to access to RSS Feeds and Podcasts in order to be always updated with the last news and information. This is a good functionality because it is possible to access to the web site’s services, not only through an Internet browser, but also through e-mail providers and mobile devices’s navigation software.

• Personnel biographies inspector: The web site provides a page dedicated to internal personnel biographies. It is a very good table (five columns) showing, sorted by name and hierarchical position, all most important people involved in defense and homeland security (civilians and military men). Every cell of the table shows the picture (121x151 pixels) and, below, the name and position. This is a good service because it is useful for lacating personnel and watching every biography by clicking the image or the text below it.

IT Defense services


Pg. 21

defense.gov biographies inspector

Watching and browsing the Italian defense web site, the user finds fewer services than the American website:

• Personnel hierarchy: The web site provides a useful service showing the internal hierarchy diagram with the Minister of Defense at vertex and all the other charges under him. Sure it is a good service but it lacks of details: for example there are no pictures and there are no names but just the charge’s title and nothing more.

• Announcements: There is a page showing all Italian military divisions and providing access to the corresponding area in order to let the user see all announcements regarding the chosen division (Es-ercito Italiano, Marina Militare Italiana or Aeronautica Italiana). This functionality is very good because it provides a fast link for other services like recruitment (a very important aspect in the context we are examinating).

• Virtual Museum: It is a functionality activated one year ago. By installing a specific 3D web engine, it is possible to download client side a VRML file showing a tridimensional museum with various items belonging to the main Italian military divisions (Esercito Italiano, Marina Militare Italiana or Aeronau-tica Italiana). It is a good interactive tool, but it is based on a not standard technology: the 3D player is the VRML player and it is not a wide diffused software (some users may decide not to install this component because they don’t trust it).

Element score

As we can see, the US Defense site has many useful services with high usability levels. The Italian Defense site does not provide so many functionalities and, furthermore, some of them does not reach a good us-ability level. Our evaluation process focuses the number of services and thei quality. For these reason, we can set these scores.



Errors managementEven when encoutering errors, web sites should pay attention to how errors are shown and what infor-mation the user is allowed to see.

Element importance

Errors management is a functionality the user does not pay attention to. It is demostrated that, when en-countering errors, the user (whether an error page is shown with some description or not) does the most common thing that everyone would do in such cases: clicking the go-back button of the browser and going to the previous page. This does not mean that a good error management system is an irrilevant element, especially for sites implementing complicated functionalities that can go wrong somewhere (for example registration procedures or transactions). For this reason we give a medium level importance to this element: Considerable (0.5).

Analyzing error management: trying to crack functionalities


Pg. 22

Examinating error management is difficult because we need to encounter an error. So our actions are just trying to crack some functionality in the application (not doing what expected) and see what we get. Lt us proceed by functionality:

• Registration form: It is possible to create, for US Defense site, a personal account indicating e-mail address and other low level profile data; if data format is not the expected one, a client validator marks incorrect fields not allowing the user to continue. The form does not perform a live email check, so if the email inserted is not correct, the user doesn’t get any advise and will never find the confirmation mail in order to complete registration. The Italian defense site does not provide a registration service.

• Unavailable resource: The most common error is the user trying to access a page that does not exist or that has been removed. So, just by typing an incorrect page, we can see how our web sites re-spond. The US Defense and IT Defense both show an error page with a simple text indicating that the requested page does not exist. However we must say that the American Defense site does not always provide the error page, in fact the application, based on ASP.NET technology for some areas, doesn’t specify the custom error page (a simple setting in the web.config file in ASP.NET architectures); if a not existing page has an extension different from “.aspx“, the error page is shown. The Italian Defense site has good settings and all calls to not existing resources make the application show the default error page.

• Multimedial errors: The US Defense site has several videos (YouTube embeeded); when data are not loaded correctly, very often because of connection problems, a text advises the user that player is unable to show content due to connection problems. The IT Defense site does not have videos or multimedia content.

Apart from every specific case, the default error pages do not provide a detailed description of the encountered problem, maybe the American site provides more textual information regarding possible causes. The Italian site provides a link, in the default error page, for returning back to the home page; the US Defense site, instead, does not provide a ruturn-to-home link, but links for going to the previous page, contacting the web site administrator and accessing the advanced site search.

Element score

There is a different approach used by both sites for showing and managing errors; in this element exami-nation, maybe, we can see a better implementation in the Italian web site (we are referring to the setting error in the US Defense site), even if the default error page in the American defense application provides more links (more usability). For these reasons we set the same score to both sites (although the American site provides videos and the registration form differently by the Italian site which is very poor from this point of view).

SCORE FOR US DEFENSE SITE IS: GOOD (2.0)


Searching in the web site


Pg. 23

Today every web site provides a search function. Some of them also implement advanced search tools to allow the user to find what he wants in the application.

Element importance

Searching in a web site having so many information and data (like US Defense and IT Defense) is essential for the user, for this reason we assign to this element the highest importance level: Determinant (1.0).

US Defense site search tool

Search tool is available to the user, at the beginning, in the top right side area of the page as a textbox with a button for submitting the search string. It is possible to perform a simple search or an advanced one by clicking the provided link near the search textbox. Adavced search has many parameters: it is possible to choose where to search (area), which words should be included and which should not be, it is possible to sort results by relevance and select also how many results showing per page; in the end there is also an option for narrowing specific elements in the results list. After inserting the string and submitting it, the results are whown as a list in Google style. There is also the possibility to perform another search process within the results using the provided link on the top and the bottom of the page. Results are paginated.

IT Defense site search tool

As for the US Defense web site the search tool is available in the same position in the page showing a textbox and a button. There are two search modes: simple and advanced search, but the advanced search is available only when a simple search is performed. The advanced search have not as many parameters as the American defense site; it is possible to type which words to search, where to search, the content type and the search type (full text or title). After clicking on the search button, the results are shown in a simple list where only the item title is available (no abstract, no narrowed words and no description is provided). The results are paginated (no possibility to set how many records show per page).

Element score

Both sites implement a search engine able to return good results basing on a string inserted by the user. Main differences, usability goals and misses rely in results quality and search options. Regarding the quality and quantity of results: both sites return a lot of elements all related to the search string; so, from this point of view, we can’t still say which search tool is better. Analyzing usability, we can notice a higher level in US Defense web site: there are more options for search process, the user can choose pagination settings and also the possibility to narrow words in the list, while all these aspects are not implemented by the Italian defense website. For this reason we can say that, for this element, both web applications reach a good score, but the US Defense web site obtains the highest one:



Functions correctnessWebsites not always implement working functions, sometimes it is possible to encounter some proce-


Pg. 24

dural errors or problems. In this final part of the functionality examination, we’ll focus on this aspect.

Element importance

Errors and inconsistance are aspects of an application that should be avoided (it is not a case that an ap-plication is always debugged before it is published/released). There is no more explaination needed to justify why we give the highest importance level to this element: Determinant (1.0).

Errors, correctness and malfunctions

In the Italian defense web site, the flash version, when using the search functionality, the final results consist in a list of numbers; probably, in the xml loaded document (the common way for interfacing Flash with a database), the programmers forgot to get from the item ID, the corresponding title and data. Fur-thermore, when clicking on an item, the Flash player tries to open a popup window for showing content, but all browsers today don’t allow popups. Regarding the US Defense site, there are no bugs or errors to report. Of course, this does not mean that our analysis has covered all pages of both applications in order to to look for malfunctions, we focused on the main areas and tried to analyze as many pages as possible. Do-ing so, this is what we got.

Element score

US Defense site passed this “user functionality test“ because we didn’t find any error. IT Defense site, in-stead, showed some problems in the Flash animated version (this implies a reduction in score).



Getting dimensional scoreUsing the same procedure of before it is possible to calculate the dimensional score for the US Defense site and the IT Defense site:


Pg. 25

( ) ( )

1 14 4

1 1 1 1

11,0 0,5 1,0 1,0 1,0 4,0 0,5 2,0 1,0 4,0 1,0 4,04,0 1,0 4,0 4,0 13,0 3,71

3,5 3,5

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + + ⋅ ⋅ + ⋅ + ⋅ + ⋅ =+ + +

= = =

∑ ∑ ∑ ∑

( ) ( )

1 14 4

1 1 1 1

11,0 0,5 1,0 1,0 1,0 1,0 0,5 2,0 1,0 3,0 1,0 3,01,0 1,0 3,0 3,0 8,0 2,28

3,5 3,5

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + + ⋅ ⋅ + ⋅ + ⋅ + ⋅ =+ + +

= = =

∑ ∑ ∑ ∑

Defense site has a higher usability level regarding the Functionality dimension; the difference with the IT Defense site is considerable.



For a better vision of the situation, we can chart data and results evaluated up until now in a bar diagram showing both scores and the mean between them.


Pg. 26


0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0

Mean

IT Defense

US Defense

When the user navigates the pages of a website, very often, he is looking for something. In many cases desired data relies in a textual information, and if text has not a good organization, the user will not be able to find what he needs so easily. It is obvious that content management is essential when we exami-nate a web application from the point of view of usability.

Content organization/labellingOrganizing and labelling content is a very important operation, doing so it is possible to individuate some areas where data and information are related. This operation let the user find what he is looking for fastly and easily.

Element importance

We have already talked about importance of this element, there no more words needed to explain why we assign to this element the highest importance level: Determinant (1.0).

US Defense content organization

The American defense web site has a good content organization. Starting from the home page to all other areas and secondary level pages, it is always possible to see a content tree somewhere with titles and brief descrition or abstracts. For example, in the home page, on the right side area, it is shown the latest news box with the main news categories in bold, and all subsections (for each category) shown in smaller characters and highlighted with a blue bright color. In the leaders page, on the left side, content is organized in a small but very useful tree, showing main categories and subsections under them. On the left side it is also possible to see the recent speeches box showing a list of the most recent speeches made by the most important US Defense leaders; speech title is bold and a brief description is provided in smaller gray characters under it. Multimedia content is also well organized. In the multimedia page it is possible to choose a section showing many photos, with their description, in a sorted list (by date) with pagination controls available on the top and bottom parts of the list. Furthermore, in every page, the content is always organized in the same way with the same graphic style; this is important because usability level increases: the user recognizes the organization and already knows how to search for what he needs.

IT Defense content organization

The Italian defense site organizes content differently. At the beginning, entering the home page, the user dosn’t find a very structured content organization. On the left side of the page it is possible to use the content menu providing an unsorted list of the main areas of the site. Sure this is one of the worst approach to content organization because there is a com-

Evaluation for Content


Pg. 27

plete lack of labelling: content is not organized in labels or groups, there is just a simple list where the user must read every item in order to find what he is looking for (if we consider that the list is also unsort-ed, searching becomes very difficult). In the other pages it is often shown a right side panel listing most recent news, but also in this case, the list is not easy to consult: every item consists in a variable length text where news title and abstract have the same character style and color; this approach is wrong because, when listing something, making the title character style different from the remeaning text, makes easier for the user to better distinguish the news itself.

Element score

Summarizing the evaluation of this element, we see very different situations in both sites, meaning that, in this place, the US Defense site reaches high usability levels while the IT Defense site is able to obtain, very difficultly, just the minimun score. This great disparity relies in the lack of a good content manage-ment in the Italian defense web application.


SCORE FOR IT DEFENSE SITE IS: VERY BAD (0.0)

Text styling and web standard styles compliance Some aspects regarding web sites text styling have already been described in the previous chapters. Here we want to examinate how both sites style textual elements. We also want to see if our wab applications are compliant to standard styling technologies like CSS; so, we are going to check W3C compliance.

Element importance

Text has a structure and an organization defined by the content manager; in order to support this struc-ture, it is possible to use advanced or even basic text styles. The higher is the styling quality, the higher is the usability level: in fact, a good text structure let the user find faster and easier what he needs. For this reason we set a medium-high importance level: Important (0,75).

Text styles

From the point of view of text styling, we have many differences between both web sites; focusing on general settings we can see different colors, different text fonts, different text sizes, different effects and layouts. By downloading CSS stylesheets of both applications, we can see what styling policy is used by the two competitors. The US Defense site’s CSS stylesheet defines Arial based font families with variable dimensions in the range 11..15 pixels. There are other elements in the stylesheet used for styling titles, subtitles, links and other textual elements in the page. It is possible to state, in general, that we have a great variety of dif-ferent textual definitions, this let the application have a good usability level because the user is able to distinguish different textual elements by their style (and different styles means different semantic refere-ments). The IT Defense site’s CSS stylesheet defines Verdana, Helvetica ans Sans-Serif based font families with 70% font size. There are a lot of definitions in the style sheet but very few of them focus on text, in fact


Pg. 28

this is the evidence explaining why the web site has not many textual styles.

W3C compliance

For a site, being compliant to standards defined by W3C sure is a good thing, also from the point of view of usability. When a site is compliant with standard technologies, it is correctly viewable by almost all browsers, this is an usability goal without any doubt. Unfortunaltely, both sites show several problems when validated using a W3C HTML validator.

• US Defense site - XHTML 1.0 Transitional: Validator finds 60 errors and 44 warnings in almost all pages; web site doesn’t pass validation.

• US Defense site - XHTML 1.0 Strict: Validator finds 63 errors and 44 warnings in almost all pages; web site doesn’t pass validation.

• IT Defense site - XHTML 1.0 Transitional: Validator finds 6 errors and 1 warning in almost all pages; web site doesn’t pass validation.

• IT Defense site - XHTML 1.0 Strict: Validator finds 12 errors and 1 warning in almost all pages; web site doesn’t pass validation.

Summarizing these results, we can say that Italian defense web site is much more compliant to CSS stan-dards.

Element score

On one side, the US Defense site has a higher usability level because of text styling, but on the other hand it fails CSS validation. The IT Defense site lives the opposite situation: CSS validation is good, but it lacks of textual styles. These considerations should justify why we assign the same score to both sites.



Information quality We have described how content should be provided and shown to the user, but what about content itself? What about the quality of provided information? This is the aspect we are going to focus on in this paragraph.

Element importance

A website can be evaluated examinating various aspects regarding usability, but one aspect of usability is the quality of information. Let us consider an example: a tourism web site describes to the user how to reach Grand Central Station starting from Central Park (North-East corner) in New York City (NY - USA). If directions we got are written in bad English, with errors, or they are just not easy to understand, we find it difficult to follow them and arriving to Grand Central, we would propably get lost and be compelled to ask for directions to someone on the road. Although a good graphic style, the content quality must be high in order to meet users’ needs, for this reason we assign to this element the highest importance level: Determinant (1.0).


Pg. 29

Evaluating content quality

Both American and Italian Defense sites have a very good information quality level. To better examinate how information is provided by the two web sites, we analyze content by section:

• News and press resources: Both sites expose content in very clear language. It is obvious that this type of information is written by journalists and reporters, for this reason, consistency and grammar correctness levels are very high. Information provided are useful and they all regard the present mili-tary situation of the considered country (USA and Italy); there are articles for every argument: from politics to military contingents spread all over the hottest world territories.

• Leaders biographies: Biographies of Italian and American leaders expose content synthetically and in the clearest way. It is possible to read events in chronological order and the user, for US leaders, can also read speeches of the most important people.

• Proceedings: Descriptions for the most important proceedings (for example how to be recruited, how to take part to special exams and so son) are explained in clear language in the common institu-tional register. Information provided are very useful for almost all kinds of users.

• FAQ: Although Italian defense web site does not provide a direct link to a FAQ area (to access this page it is necessary to search the word: “FAQ“), both web sites have good Frequently Asked Questions pages. IT Defense site provides a list with some questions (bold text) and the corresponding answer (plain text) regarding post military service employment; descriptions are not much detailed but in some answer, a link to a detail page is provided. US Defense site has a much better page dedicated for Questions. It is possible to access three areas, where the user can browse questions and answers, or making new questions adding some comments or even accessing to personal questions made before. Questions and answers are styled differently and they are paginated, it is also possible to search for a question or search by argument too. Provid-ed content is very self-explaining and interactive, clicking on one question, not only the answer, but also other information and links to other questions related to the chosen one are shown.

Element score

There is not much disparity between the two sites, we just need to point out that the US Defense site reaches a better usability level because of the FAQ section having more options and more information and data.



Website localization: multilanguage supportThere are websites, operating in some particular contextes, that must provide support for more than one language. In our case we are operating inside the institutional/military context: does a web site operat-ing here need to provide support for other languages? The answer is yes and no. Our web sites (US and IT defense) provide information to Italian and American people about military contigents, military opera-tions, politicians and so on, but all these data regard the home territory, and in the home territory there is


Pg. 30

no need to have a different language from the local one. So at a first look we would state that a different language is useless to be provided, but it would not be correct; there are users accessing the web site other than citizens or American/Italian people: for example journalists, researchers, foreign people and so on; they would probably like to read information provided in both sites.

Element importance

It is important to focus that an additional language is not such a strong requirement, our sites must first provide a service for local people (USA or Italian citizens), and then, eventually, provide additional ser-vices for other languages. For this reason we assign to this element a medium importance level: Consider-able (0.5).

Localization in IT Defense and US Defense sites

There is nothing much to say about the US Defense web site, because there are no other languages other than American in every page of the application. The Italian defense web site provides support for English, French and German but in some pages only, for example, the most important leaders’ biographies are available in four languages (including Italian).

Element score

This is the first element where IT Defense site obtains a higher score than US Defense.

SCORE FOR US DEFENSE SITE IS: VERY BAD (0.0)


Getting dimensional scoreUsing the same procedure of before it is possible to calculate the dimensional score for the US Defense

site and the IT Defense site. Defense site has a higher usability level regarding the Content dimension; the difference with the IT De-fense site is considerable. Note that in this dimension both sites obtain a very low score, this is because there are elements where they got a 0.0 usability level.


Pg. 31

( ) ( )

1 14 4

1 1 1 1

11,0 0,75 1,0 0,5 1,0 4,0 0,75 2,0 1,0 4,0 0,5 0,04,0 1,5 4,0 0,0 9,50 2,92

3,25 3,25

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + + ⋅ ⋅ + ⋅ + ⋅ + ⋅ =+ + +

= = =

∑ ∑ ∑ ∑

( ) ( )

1 14 4

1 1 1 1

11,0 0,75 1,0 0,5 1,0 0,0 0,75 2,0 1,0 3,0 0,5 2,00,0 1,5 3,0 1,0 5,50 1,69

3,25 3,25

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + + ⋅ ⋅ + ⋅ + ⋅ + ⋅ =+ + +

= = =

∑ ∑ ∑ ∑



Charting results it is possible to see the global situation from a better point of view. The bar diagram shows both scores and the mean between them.


Pg. 32


0,0 0,5 1,0 1,5 2,0 2,5 3,0

Mean

IT Defense

US Defense

A web site can also be evaluated considering how well it is managed by developers, content managers and web masters. The better is managed a web site, the better will be the usability level.

Site availabilityUsers accessing a web site sometimes are not able to reach the entire application or some part of it due to unavailability situations. When a web site is unavailable (or even some part of it), the user cannot ac-cess its services, this is sure a usability failure.

Element importance

Keeping available a web application is essential and very important, the user should always access al ser-vices, this depends by many factors, one is, for example: maintainance. Importance level is: Determinant (1.0).

Availability analysis

It is much difficult to see if a web site is always available. Unavailability situations can be noticed when a web application is navigated day by day; in our case, we need much more time. Not having this time, we perform a different operation: looking into forums we questioned people about the US Defense web site availability and the Italian defense site availability. This approach returned some results only for the American site (good results: the site is always reachable); we could not retrieve any information about the IT Defense site in forums and chats; for this reason we can rely only on our visits and, doing so, no prob-lems were noticed.

Element score

Although this element could not be processed properly, we assign a score to it basing on data we could retireve; given this situation we assign to both sites the highest score.


SCORE FOR IT DEFENSE SITE IS: EXCELLENT (4.0)

Site monitoringThis is another element very difficult to evaluate because we need to be part of the development team of both sites. We obviously cannot meet this condition and, in order to assign a score, we can try to exami-nate this element operating differently.

In this case we sent to both sites’ webmasters an email, asking for information about site monitoring ex-plaining our intentions (a site analysis for academic and research purposes). Although we got an answer

Evaluation for Site Management


Pg. 33

by the two webmasters, they didn’t provide us any information regarding sensible data.

Element score

We cannot assign a score because of what said before. When evaluating the dimensional score, we’ll not take into consideration this element.

SCORE FOR US DEFENSE SITE IS: ??

SCORE FOR IT DEFENSE SITE IS: ??

Site updatingA constantly updated web site sure provides high quality services, especially when the web application exposes pages and areas like news or press releases.

Element importance

Our applications provide a lot of content that needs to be updated daily, for this reason this element is assigned to a high importance level: Determinant (1.0).

Evaluating updating status

Updating status is a term used for referring to the hightest updating frequency of a site, it is like answer-ing to the question: “How many times per day do you update the web site?“. In our case we must try to obtain this data by looking at the web sites and seeing what changes everyday. Focusing on the US Defense website we can notice, navigating everyday the web application, a high updating level: the highlight images change every three or four hours , the main news are shown in rota-tor mode so that every time the user can read the title of a different news, even if the set of news doesn’t change in the same day. This pattern is almost the same in the pages providing access to the main areas of the site, a set of images is shown in these pages and they rotate thanks to an interactive tool (Flash player), the next day it is possible to see a different set of images. Also the textual content changes sometimes in the most important areas, but somewhere it is static (for example in the pages dedicated to leaders biographies). Moving to the IT Defense site, things are very different and much more static. The web site, in fact, is updated only in the news area; even the home page is not changed, the only image (the central news im-age) has remained the same for days.

Element score

It is easy to assign a score for this element given the high updating level of the US Defense site and the low level of the Italian defense site.



Broken links


Pg. 34

Element importance

Broken links represent a very common situation the user must face when navigating a web site in the Internet, when a web application expose links that refer to non reachable locations, the result is an error page (if present) avising the encountered problem, the user so must go back and choose another link (hoping that it works). Such situations should be avoided by web developers, for this reason the impor-tance level for this element is high: Important (0.75).

Searching for broken connections

It is very strange to say and even unexpected but, examinating the main and the most important areas of both sites, we were not able to detect any broken kink. All links work fine.

Element score

Given our considerations, there is no need to not assign the highest score to both web sites.



Contacts and communication with usersGiving to users the possibility to communicate with web site managers or masters is a good thing. This let the user find some explainations in the most difficult procedures in the application, and anyway, when the user can communicate with someone on the other side, he perceives a good feedback and his opinion on the web site is postive.

Element importance

Communicating with users and letting them contact someone behind the web application is an impor-tant usability aspect, that’s why the importance level for this element is the highest: Determinant (1.0).

How can I communicate with someone behind the web application?

When trying to communicate with someone inside the US Defense web site, things get very easy. It is unexpected such a well organized communication structure, but just accessing the “Contact Us“ section of the site, it is possible to see a lot of options. The first step is locating the right link, in fact, depending on who you are and what you seek, there is the specific link with the right information you need to con-tact someone. There are links for every kind of information the user may ever need: links for journalists or normal users, links for asking information to Department of Defense personnel or Petagon personnel; it is also possible contacting personnel in the US Army, Navy, Marine Corps and Air Force, and there are links even for veterans and retired civilians. But it’s not finished here. When writing an email (using often a form provided by the web site, rather than using your own email composer) to any provided contact, an autogenerated message appears in you mailbox at once, telling you that your email has been processed and that very soon an operator will send you a message trying to answer your questions or providing the information you asked; and after few hours the promised email comes.

Moving to the Italian Defense site, everything changes. Unfortunately, there is not much to say: the web site does not provide any informatic way for contacting someone behind the site. There is not an email


Pg. 35

or a commnunity web site, there are just links to other sites (and even there, the user finds few emails) and lots of phone numbers. This represents a very bad situation becuse the user cannot contact anybody utulizing his own computer.

Element score

For this element, the US Defense site undoubtly reaches the hight score, while the IT Defense site doesn’t meet any requirement, for this reason, score is zero.


SCORE FOR IT DEFENSE SITE IS: VERY BAD (0.0)

Getting dimensional scoreUsing the same procedure of before it is possible to calculate the dimensional score for both sites:

The American Defense site has a higher usability level regarding the Management dimension.




Pg. 36

( ) ( )

1 14 4

1 1 1 1

11,0 1,0 0,75 1,0 1,0 4,0 1,0 4,0 0,75 4,0 1,0 4,04,0 4,0 3,0 4,0 15,0 4

3,75 3,75

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + + ⋅ ⋅ + ⋅ + ⋅ + ⋅ =+ + +

= = =

∑ ∑ ∑ ∑

( ) ( )

1 14 4

1 1 1 1

11,0 1,0 0,75 1,0 1,0 4,0 1,0 1,0 0,75 4,0 1,0 0,04,0 1,0 3,0 0,0 8,0 2,46

3,25 3,25

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + + ⋅ ⋅ + ⋅ + ⋅ + ⋅ =+ + +

= = =

∑ ∑ ∑ ∑


0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0

Mean

IT Defense

US Defense

We are going to evaluate both applications from the point of view of accessibility in order to see how much reachable are the web sites or if disabled people can navigate the applications.

Site reachabilityWhen we look for a web site, if we don’t know the site path or complete name, we try to use a web search engine in order to find it using some keywords. In our case these keywords would probably be: Ameri-can defense, Italian defense, Ministro della difesa, Secretary of defense, defense, difesa, military defense, difesa militare, homeland security, sicurezza nazionale and so on. Do the two sites appear among the first six or ten entries of the final results of the most common and famous search engines?

Element importance

This element is very important because people who never visited both sites, can reach them without even knowing their complete names; this represents sure a good usability aspect. Importance level is: Determinant (1.0).

Searching for US Defense and IT Defense sites

To test reachability for both sites, let us take into consideration these search keywords:

• Keywords K1: American, Defense | Difesa, Italiana

• Keywords K2: USA, Homeland, Security | Sicurezza, Nazionale, Italiana

• Keywords K3: USA, Department, Defense | Ministero, Difesa, Italiana

• Keywords K4: USA, Secretary, Defense | Ministro, Difesa, Italiana

• Keywords K5: USA, Military, Defense | Difesa, Militare, Italiana

For every group of keyword (KX) we will perform a search using one of these engines: Google, Microsoft Bing, AltaVista, Yahoo Search and InfoSeek Japan. Testing every group of words, we’ll annotate the posi-tion where the web site appears in order to understant its reachability.

After doing as many searches as for the number of keyword groups (for every web search engine), we’ll build a summarization table showing on rows the search engines, and on columns the keyword groups using KX notation. In every cell it is shown the positions where the web site appears, we can have one position or more de-pending on how many entries the web site is visualized in the results list.

Let us first take into consideration the US Defense site:

Evaluation for Accessibility


Pg. 37

K1 K2 K3 K4 K5

GOOGLE Not found Not found 1st*; 3rd 1st*; 3rd; 13th* 1st*; 4th

MS BING Not found Not found 2nd*; 3rd; 7th* 1st; 4th*; 6th 5th*; 12th

ALTAVISTA Not found Not found 2nd; 3rd*; 5th* 2nd*; 3rd; 4th* 3rd; 4th*; 9th*

YAHOO Not found 33rd* 1st*; 2nd*; 3rd 3rd; 4th; 6th* 1st*; 4th; 5th

INFOSEEK (JP) Not found Not found 17th* Not found Not found

Note: for elements with * it means that shown link adresses the user to the subdomain www.defenselink.mil used by www.defense.gov as a support domain.

Moving now to the Italian defense site:

K1 K2 K3 K4 K5

GOOGLE 1st; 2nd*; 3rd* Not found 1st; 2nd*; 3rd* 1st; 2nd*; 3rd*1st*; 2nd*;

3rd*

MS BING 1st; 2nd*; 6th Not found 1st; 3rd*; 7th 1st; 3rd* 1st*; 2nd; 6th*

ALTAVISTA 1st; 2nd; 6th* 5th 1st; 3rd; 4th* 1st; 3rd 1st; 2nd; 3rd*

YAHOO 1st Not found 1st; 3rd* 1st 1st*; 5th*; 9th

INFOSEEK (JP) 1st Not found 1st; 2nd; 3rd* 1st1st*; 2nd*;

6th*

Note: for elements with * it means that shown link adresses the user to a subdomain of www.difesa.it for ex-ample: www.esercito.difesa.it or www.marina.difesa.it and so on.

Element score

This is an unexpected result: the Italian defense web site is more reachable (even in the Japanese area) than the American one. American defense gets 14 matches while Italian defense gets 21 matches in all search engines.



Browser compatibilityIf someone questioned a web developer: “What’s the thing you are most worried about?“, the poor developer would probably answer you: “Web browsers“. It is simple, having a web site compatible with all browsers (meaning a site that can be viewed in the same identical way indipendently by the browser utilized) is a great usability goal. All over the world there are more than 7.5 billions people and they sure don’t use the same browser; letting a site be visualized almost in every browser is very difficult, but if this requirement is satisfacted, usability level increases a lot.


Pg. 38

Element importance

This element is very important because people don’t use the same browsers. Given this situation, it is necessary to build web sites trying to let different users see always the same graphic structure. For this reason we assign the highe importance level: Determinant (1.0).

How are viewed our websites in the most famous browsers in the world?

We must try to be clear and examinate every site’s characteristic in some browsers. We choose these ones:

• Microsoft Internet Explorer 8.0

• Mozilla FireFox 3.0

• Apple Safari 4.0

The characteristics we want to analyze regard the most common graphical aspects: tables, divs, fonts, form controls and so on.

Let us start with the American defense site:

INTERNET EXPLORER FIREFOX SAFARI

TABLES MANAGEMENT Correct view Correct view Correct view

DIVS MANAGEMENT Correct viewSome divs overlap on

others hiding some links and text

Some divs overlap on others hiding some

links and text

FONTS AND TEXT Correct view

Text aligned to image is rendered differently, not respecting condi-

tions on margins

Correct view

LINKS Correct view Correct view Correct view

FORM CONTROLS Correct view Correct view Correct view

MULTIMEDIA Correct view Correct view Correct view

BACKGROUNDS Correct view Correct view Correct view

Now let us move to the Italian defense site:


TABLES MANAGEMENT Correct view Correct view Correct view

DIVS MANAGEMENT Correct view Correct view Correct view


Pg. 39


FONTS AND TEXT Correct view

Text margins and text wrap are different. Bold characters are greater.

Some fonts are ren-dered differently.

Correct view

LINKS Correct view Correct view Correct view

FORM CONTROLS Correct viewBorders are rendered

differently (solid and no 3d effect like others)

Correct view

MULTIMEDIA Correct view Correct view Correct view

BACKGROUNDS Correct view Correct view Correct view

Element score

Well, from the point of view of browser compatibility, both sites have few problems with FireFox. Apple and MSIE view the sites almost in the same way. For this reason we assign to both applications the same score.



AccessibilityHere’s a very huge problem for many web sites. Today, providing site accessibility for disabled people has become a very important requirement, but not all web sites provide services in this direction. There are many ways for making a web application usable also for people that have not our same skills; in our case we’ll check for our web applications to be compliant to some simple standards, in particular those ones:

• WAI/WCAG: They are guidelines defined by W3C and they are still working on it.

• Vocal synthesis guidelines: There are special programs that use vocal synthesis for speaking textual content of Internet pages; in order to let these services work on web pages, it is necessary to pay attention to few things, even taking care of small details it is possible to provide a better service, for everyone, for real.

Element importance

We assign to this element the highest importance level: Determinant (1.0).

Accessibility validation

We want to evaluate both sites using some guidelines obtained by merging WAI and other important guidelines:

• Color blindness: Some people cannot see colors in the same way like everyone because of many


Pg. 40

color deficiencies. It is possible to pay attention in web sites, and using good colors in order not to confuse those people affected by this inability. In this place we’ll focus on two important known defi-ciencies: Protanopia and Deuteranopia, two forms of Dichromacy, the inability of distinguishing two close colors placed in the same chromatic range (red, green or blue). In order to avoid this problem it is required not to place two similar colors togheter in the same page, for example the orange near to red, if these colors are used to distinguish two different types of information, the user, affected by Dichromacy, will not notice the difference.

• Partially sighted and blind people: There are people who cannot see and people who can see but very difficultly; life for them is much difficult because they don’t have the most used sense: sight. In order to let them use special programs based on vocal synthesis (text to sound), web sites should respect web standards and following these rules:

• When listing items, use <ol> or <ul> and <li> tags.

• Use always CSS to format text (at every level: external file, page level or inline level).

• Use spans and divs, and in every image always remember to place the alt attribute.

• Do not use tables for defining pages layouts.

• When defining top bars or menus, use HTML horyzontally or vertically styled list tags.

• Use attribute alt wherever it is allowed.

• Do not use multimedial components (for example Flash player) for showing important infor-mation.

• Do not use images for defining buttons or creative content.

• Provide every anchor <a> with the attributes rel and title correctly set.

These few rules can help building accessible web sites. We are going to check whether our web applica-tions are accessible by all types of users, by following these guidelines.

Let us focus on the American and Italian defense web sites:

DEFENSE.GOV DIFESA.IT

COLORS COMBINATION

Blue background and white fore-ground is a good combination but

the top bar combines different shades of blue (it is not a great prob-

lem because the white text gener-ates a very good constrast after all).

Background/Foreground combina-tion is absolutely wrong because

these colors will be merged by people affected by Dichromacy.

Generally there is a very bad choice of colors, the entire page itself is flat-tened when seen by people affected

by color blindness because of the lack of contrast: there is a unique

color range: gray/light blue.


Pg. 41

DEFENSE.GOV DIFESA.IT

HTML LISTS OK OK

CSS STYLES OK OK

SPANS AND DIVS OK OK

TABLES

The top area (background top im-age, logo and search textbox) is

structured using a table. This could be avoided using divs.

OK

MENUES AND TOP BARS

OK OK

IMAGES OK OK

MULTIMEDIA

In the home page there is an images rotator, apparently it might seem a Flash component, but it is a javas-

cript/html ASP.NET ASHX control: it does not create any problem.

There are no multimedial compo-nents.

BUTTONS OK OK

LINKS Rel attribute is not specified in links. Rel attribute is not specified in links.

Element score

Although final result were expected to be not very good, we must say the contrary. Both sites have a good usability level for disabled people even if some rule is not respected, we reach good scores.



Getting dimensional scoreUsing the same procedure of before it is possible to calculate the dimensional score for both sites:

The IT Defense site has a higher usability level (it is the first time) regarding the Accessibility dimension.




Pg. 42

( ) ( )

1 13 3

1 1 1 1

11,0 1,0 1,0 1,0 2,0 1,0 3,0 1,0 3,02,0 3,0 3,0 8,0 2,66

3,0 3,0

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + ⋅ ⋅ + ⋅ + ⋅ =+ +

= = =

∑ ∑ ∑ ∑

( ) ( )

1 13 3

1 1 1 1

11,0 1,0 1,0 1,0 4,0 1,0 3,0 1,0 3,04,0 3,0 3,0 10,0 3,33

3,0 3,0

n n

i i i i i ii i i i


= = = =

−

= ⋅ ⋅ = ⋅ ⋅ =

= + + ⋅ ⋅ + ⋅ + ⋅ =+ +

= = =

∑ ∑ ∑ ∑

This is the first time that the Italian defense site gets a higher score than the American one.


Pg. 43


0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5

Mean

IT Defense

US Defense

We are now ready to resume every dimension analyzed up until now in a single chart and in a single number.

Global evaluation

Global diagram for US Defense web site Global diagram for IT Defense web site

The chart on the left shows a global vision (all dimensions) for the US Defense site. Almost all dimensions have good levels, but we can see that Local-ization is the Achilles’ heel of this application. Regarding everything else, there are no problems at all.

The chart on the right shows a global vision (all dimensions) for the IT Defense site. If compared to the US defense chart, we can see a totally different condition. Almost all dimensions have low levels, there are just two dimensions that reach the highest level. Generally speaking, usability is not good.


Pg. 44

0,51,01,52,02,53,03,54,0

Accessibility

Management

Content

Functionality

Communication

Architecture

0,5

1,0

1,5

2,0

2,5

3,0

3,5

Accessibility

Management

Content

Functionality

Communication

Architecture

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0Accessibility

CompatibilityReachability

Contacts

Broken links

Updating

Availability

Localization

Info qualityText styles Labelling

CorrectnessSearch

Errors

Functions

Graphics

Brand image

Home page

External linksSite map

NavigationSite structure

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

AccessibilityCompatibility

Reachability

Contacts

Broken links

Updating

Availability

LocalizationInfo quality

Text stylesLabelling

CorrectnessSearch

Errors

Functions

Graphics

Brand image

Home page

External links

Site mapNavigation

Site structure

As announced before, we would like to get two numbers in order to make the simplest mathematical inequality. To do so, we need to use the formula introduced in the first chapter.

Getting the final numbers

BEING 3,44 > 2,23 WE CAN FINALLY STATE THE US DEFENSE WEB SITE IS MUCH BETTER THAN THE IT DEFENSE WEB SITE.


Pg. 45

22

1 1

22

US1 1

1,0 1,0 0,75 0,75 1,0 0,2 1,0 1,0 0,5 1,0

1,0 1,0 0,75 1,0 0,5 1,0 1,0 0,75 1,0 1,0 1,0 1,0 19,21 0,0520

1,0 3,0 1,0 3,0 0,75 4,0 0,75 4,0 1,0 4

0,0520

n

i ii i

n

i i i ii i

T P P

S P E P ET

= =

= =

= = = + + + + + + + + + +

+ + + + + + + + + + + + =

= ⋅ ⋅ = ⋅ ⋅ =

⋅ + ⋅ + ⋅ + ⋅ + ⋅

= ⋅

∑ ∑

∑ ∑,0 0,2 4,0

1,0 4,0 1,0 4,0 0,5 2,0 1,0 4,0 1,0 4,0 1,0 4,00,75 2,0 1,0 4,0 0,5 0,0 1,0 4,0 1,0 4,00,75 4,0 1,0 4,0 1,0 2,0 1,0 3,0 1,0 3,0

3,0 3,0 3,0 3,0 4,0 0,8 4,0 4,0 1,00,0520

+ ⋅ + + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + = + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + + ⋅ + ⋅ + ⋅ + ⋅ + ⋅

+ + + + + + + + += ⋅

( )

22

IT1 1

4,0 4,04,0 1,5 4,0 0,0 4,0 4,0 3,0 4,0 2,0 3,0 3,0

0,0520 12,0 8,8 13,0 9,5 15,0 8,00,0520 66,3 3,44

1 0,0520

1,0 3,0 1,0 1,0 0,75 2,0 0,75 2,0 1,0 2,0 0,

0,0520

n

i i i ii i

S P E P ET = =

+ + = + + + + + + + + + + +

= ⋅ + + + + + == ⋅ =

= ⋅ ⋅ = ⋅ ⋅ =

⋅ + ⋅ + ⋅ + ⋅ + ⋅ +

= ⋅

∑ ∑2 2,0

1,0 2,0 1,0 1,0 0,5 2,0 1,0 3,0 1,0 3,0 1,0 0,00,75 2,0 1,0 3,0 0,5 2,0 1,0 4,0 1,0 1,00,75 4,0 1,0 0,0 1,0 4,0 1,0 3,0 1,0 3,0

3,0 1,0 1,5 1,5 2,0 0,4 2,0 1,0 1,0 3,0 30,0520

⋅ + + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + = + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + + ⋅ + ⋅ + ⋅ + ⋅ + ⋅

+ + + + + + + + + += ⋅

( )

,00,0 1,5 3,0 1,0 4,0 1,0 3,0 0,0 4,0 3,0 3,0

0,0520 7,0 4,4 8,0 5,5 8,0 10,00,0520 42,9 2,23

+ = + + + + + + + + + + +

= ⋅ + + + + + == ⋅ =

comparative analysis on e-gov web sites

Technology

services web sites

siderarionsboth web

main services

web sitesrepresenting

country defense institutions

country defense system

country defense headquarters

country defense departments