Analysis of NASA's Earth Observing System Data Information (EOSDIS) Core System- The ECS Project

Saheed. A. Busari and Emmanuel Letier

RADAR is a lightweight modelling language and tool for analysing software requirements and architecture decisions (Busari and Letier, 2016). This report presents an application of RADAR to the analysis of the NASA's Earth Observing System Data Information (EOSDIS) Core System, also known as the ECS Project.

Our analysis is motivated by and based on previous study in the 2002 SEI report on "Making Architecture Design Decisions: An Economic Approach" (Kazman et al., 2002). Our model objectives, decisions and data were obtained from 2002 SEI report.

Problem Description

The ECS is a large scale distributed data information system used in managing and distributing huge volumes of climate related data in different forms around the world, 24 hours each day. This system collects and manages more than 1000 gigabytes of data, every day, from several satellites using various sensors (Kazman et al., 2001,Kazman et al., 2002 ).

As reported by Kazman et al., 2002, the ECS went through a maintenance phase and a planning process of boosting its capabilities. However, the ECS project manager had limited annual budget to maintain and enhance the functionality of the system. In a bid to achieve this goal, proir analysis, using the ATAM methodology, was performed to identify a set of architectural strategies, which we call decisions in our model (as shown in figure 1), to be made in enhancing the system. The manager is faced with selecting among the set of decisions that give maximum utilty in the project and also minimise the project cost.

Note: The Cost Benefit Analysis Method (CBAM) (Kazman et al., 2001,Kazman et al., 2002) was used in the 2002 SEI report, and it estimates individual impact of architectural strategies on utility, but not their combined impacts. However, our model analyses the interaction between architectural strategies ( i.e the combined impact of architectural strategies), and allows the selection of more than one architectural strategy.

Figure 1: Design Decisions (click on the image to zoom in)

Optimisation Objectives

Objective Max ExpectedUtility = EV(Utility);
Objective Min Cost;

Modelling Utility

      
Utility =   HungRequestsRatio_weight * HungRequestsRatio_Utility +
            LostRequestsRatio_weight * LostRequestsRatio_Utility +
            FailedOrderRatio_weight * FailedOrderRatio_Utility +
            HungOrdersRatio_weight * HungOrdersRatio_Utility +
            LostOrdersRatio_weight * LostOrdersRatio_Utility +
            HelpNeededByUsers_weight * HelpNeededByUsers_Utility +
            FailureInfoGivenToUsers_weight * FailureInfoGivenToUsers_Utility +
            LimitOnOrders_weight * LimitOnOrders_Utility +
            NotificationsFrequency_weight * NotificationsFrequency_Utility +
            Performance_weight * Performance_Utility
  
HungRequestsRatio_weight = deterministic(10);
LostRequestsRatio_weight = deterministic(15);
FailedOrderRatio_weight = deterministic(15);
HungOrdersRatio_weight = deterministic(10);
LostOrdersRatio_weight = deterministic(15);
HelpNeededByUsers_weight = deterministic(10);
FailureInfoGivenToUsers_weight = deterministic(5);
LimitOnOrders_weight  = deterministic(5);
NotificationsFrequency_weight = deterministic(10);
Performance_weight = deterministic(5);

Hung Request Ratio Utility

HungRequestRatio_Utility = (HungRequestRatio - HungRequestRatio_worst)/(HungRequestRatio_best - HungRequestRatio_worst)
HungRequestRatio_best =deterministic(0);
HungRequestRatio_worst = deterministic(10%);

Lost Requests Ratio Utility

LostRequestsRatio_Utility = (LostRequestsRatio - LostRequestsRatio_worst)/(LostRequestsRatio_best - LostRequestsRatio_worst)
LostRequestsRatio_best = deterministic(0);
LostRequestsRatio_worst = deterministic(5%);

Failed Order Ratio Utility

FailedOrderRatio_Utility = (FailedOrderRatio - FailedOrderRatio_worst)/(FailedOrderRatio_best - FailedOrderRatio_worst)
FailedOrderRatio_best =deterministic(0);
FailedOrderRatio_worst = deterministic(10%);

Hung Orders Ratio Utility

HungOrdersRatio_Utility = (HungOrdersRatio - HungOrdersRatio_worst)/(HungOrdersRatio_best - HungOrdersRatio_worst)
HungOrdersRatio_best = deterministic(0);
HungOrdersRatio_worst = deterministic(10%);

Lost Orders Ratio Utility

LostOrdersRatio_Utility = (LostOrdersRatio - LostOrdersRatio_worst)/(LostOrdersRatio_best - LostOrdersRatio_worst)
LostOrdersRatio_best =deterministic(0);
LostOrdersRatio_worst = deterministic(10%);

Help Needed By Users Utility

HelpNeededByUsers_Utility = (HelpNeededByUsers - HelpNeededByUsers_worst)/(HelpNeededByUsers_best - HelpNeededByUsers_worst)
HelpNeededByUsers_best = deterministic(0);
HelpNeededByUsers_worst = deterministic(50%);

Failure Info Given To Users Utility

FailureInfoGivenToUsers_Utility = (FailureInfoGivenToUsers - FailureInfoGivenToUsers_worst)/(FailureInfoGivenToUsers_best - FailureInfoGivenToUsers_worst)
FailureInfoGivenToUsers_best = deterministic(100%);
FailureInfoGivenToUsers_worst = deterministic(10%);

Limit On Orders Utility

LimitOnOrders_Utility = (LimitOnOrders - LimitOnOrders_worst)/(LimitOnOrders_best - LimitOnOrders_worst)
LimitOnOrders_best = deterministic(0%);
LimitOnOrders_worst = deterministic(50%);

Notifications Frequency Utility

NotificationsFrequency_Utility = (NotificationsFrequency - NotificationsFrequency_worst)/(NotificationsFrequency_best - NotificationsFrequency_worst)
NotificationsFrequency_best = deterministic(1);
NotificationsFrequency_worst = deterministic(1/1000);

Performance Utility

Performance_Utility = (Performance - Performance_worst)/(Performance_best - Performance_worst)
Performance_best = deterministic(90%);
Performance_worst =  deterministic(50%);

Figure 2: Utility AND/OR graph: click on the image to zoom in

Modelling Quality Attributes

Hung Request Ratio

HungRequestRatio = HungRequestRatio_current 
                    * (1 - ReassignedHungRequestRatio) 
                    * (1 - ForcedHungRequestRatio)
                    
HungRequestRatio_current = deterministic(5%);
                    
ReassignedHungRequestRatio = decision("Order Reassignment"){
        "Current: not possible to reassign order" : deterministic(0);
        "Allow Order Reassignment": deterministic(60%);
}   

ForcedHungRequestRatio = decision("Forced Order Completion"){
        "Current: not possible to force order completion" : deterministic(0);
        "Allow Forced Order Completion": deterministic(40%);
}                   

Lost Request Ratio

LostRequestRatio = triangular(0, 0.5, 1);

Failed Order Ratio

FailedOrderRatio = decision("Order Persistence Strategy"){
        "Current: store when processed" :  deterministic(5%);
        "Store as soon as received"      : deterministic(2%);
}

Hung Orders Ratio

HungOrdersRatio = HungOrdersRatio_current *
                    (1 - SkippedHungOrderRatio) *
                    (1 - RetriedHungOrderRatio)

HungOrdersRatio_current = deterministic(10%);
                    
SkippedHungOrderRatio = decision("Order Segmentation"){
        "Current: no order segmentation" : deterministic(0);
        "Orders are segmented"            : deterministic(60%);
}

RetriedHungOrderRatio = decision("Hung Order Recovery"){
        "Current: no order retry"   : deterministic(0);
        "Allow Order Retry"         : deterministic(40%);
}

Lost Orders Ratio

LostOrdersRatio = decision("Order Persistence Strategy"){
        "Current: store when processed" : deterministic(1%);
        "Store as soon as received"      : deterministic(0%);
}

Help Needed by Users

HelpNeededByUsers = LostOrdersRatio * HelpNeededPerLostOrder * HelpNeededForTrackingGranularity

HelpNeededPerLostOrder = decision("Failed Order Notification"){
        "Current: no notification":deterministic(25%);
        "User notified of failed order":deterministic(20%);
}

HelpNeededForTrackingGranularity = decision(Order Tracking){
        "Current: order level"          : deterministic(1);
        "Granule-level order tracking"  : deterministic(40%)
 }

Failure Information

FailureInfoGivenToUsers = 1- RatioUsersNotGettingInfo
RatioUsersNotGettingInfo = RatioUsersNotGettingInfo_current * 
                            (1 - NotificationOrderEffect) * 
                            (1 - OrderTrackingGranularityEffect) *
                            (1 - UserInformationEffect)
                            
RatioUsersNotGettingInfo_current = deterministic(50%);

NotificationOrderEffect = decision("Failed Order Notification"){
        "Current: no notification": deterministic(0);
        "User notified of failed order":  deterministic(80%);
}

OrderTrackingGranularityEffect = decision(Order Tracking){
        "Current: order level"          : deterministic(0);
        "Granule-level order tracking"  : deterministic(90%);
 }
 
UserInformationEffect = decision("Available User Information"){
        "Current: no link to user info"     : deterministic(0);
        "Link to user information"          : deterministic(20%);
}

Limit on Order

LimitOnOrders = decision("Order Chunking"){
        "Current: no oder chunking": deterministic(30%);
        "Order Chunking"            : deterministic(15%);
}

Notification Frequency

NotificationFrequency = decision("Order Bundling"){
        "No Order Bundling" : deterministic(1);
        "Order Bundling"    : deterministic(0.01);
}

Performance

Performance = decision("Order Bundling"){
        "No Order Bundling" : deterministic(60%);
        "Order Bundling"    : deterministic(55%);
}

Modelling Cost

Cost = Cost_OrderReassignment +
        Cost_ForcedOrderCompletion +
        Cost_OrderPersistenceStrategy +
        Cost_OrderSegmentation +
        Cost_HungOrderRecovery +
        Cost_FailedOrderNotification +
        Cost_OrderTracking +
        Cost_AvailableUserInformation +         

Cost_OrderReassignment = decision("Order Reassignment"){
        "Current: not possible to reassign order" : deterministic(0);
        "Allow Order Reassignment":  triangular(360, 400, 440);
}   

Cost_ForcedOrderCompletion = decision("Forced Order Completion"){
        "Current: not possible to force order completion" : deterministic(0);
        "Allow Forced Order Completion":  triangular(180, 200, 220);
}

Cost_OrderPersistenceStrategy = decision("Order Persistence Strategy"){
        "Current: store when processed" : deterministic(0);
        "Store as soon as received"      : triangular(1200, 1200, 7150);
}

Cost_OrderSegmentation = decision("Order Segmentation"){
        "Current: no order segmentation" :  deterministic(0);
        "Orders are segmented"       : triangular(180, 200, 220);
}

Cost_HungOrderRecovery = decision("Hung Order Recovery"){
        "Current: no order retry"   : deterministic(0);
        "Allow Order Retry"     :triangular(180, 200, 220);
}

Cost_FailedOrderNotification = decision("Failed Order Notification"){
        "Current: no notification": deterministic(0);
        "User notified of failed order": triangular(270, 300, 330);
}

Cost_OrderTracking = decision(Order Tracking){
        "Current: order level"      :  deterministic(0);
        "Granule-level order tracking"  : triangular(900, 1000, 1650);
}

Cost_AvailableUserInformation = decision("Available User Information"){
        "Current: no link to user info"     : deterministic(0);
        "Link to user information"      :  triangular(90, 100, 440);
}

Cost_OrderChunking = decision("Order Chunking"){
        "Current: no oder chunking" : 0
        "Order Chunking"        : triangular(360, 400, 440);
}

Cost_OrderBundling = decision("Order Bundling"){
        "No Order Bundling"     :  deterministic(0);
        "Order Bundling"    :triangular(360, 400, 440);
}       
            

Figure 3: Cost AND/OR graph: click on the image to zoom in

Analysis Results

Optimisation Analysis

Here is a summary of the modelling and analysis information:

The model analysis generates the following shortlist of Pareto-optimal solutions:

Figure 4: ECS pareto front

These results show that all shortlisted solutions include the options Order Chunking , Orders are segmented and No order Bundling. This means that, in our model, these three options, respectively, outperform the options Current: No Order Chunking, Current: No Order Segmentation and No Order Bundling on both objectives.

But once these three options are selected, the shortlist includes different combinations of Order Reassignment, Forced Order Completion, Order Persisitence Strategy, Hung Order recovery, Failed Order Notification, Order Tracking, Available User Information; each combination representing a different tradeoffs between maximising ExpectedUtility and minimising Cost.

To visualise such tradeoffs, RADAR generates the graph in Fig. 4 plotting the objective values for the shortlised solutions (shown squares at the top of the graph) and all other non shortlisted ones (shown as circles).

Information Value Analysis

Analysing information value gives the following results that show that the information value for all model parameters is very low:

EVTPI = 0

References

1. Saheed A. Busari and Emmanuel Letier, "RADAR: A lightweight tool for requirements and architecture decision analysis," in submission, August 2016

2. Kazman, Rick, Jai Asundi, and Mark Klein. "Quantifying the costs and benefits of architectural decisions." Proceedings of the 23rd international conference on Software engineering. IEEE Computer Society, 2001.

3. Moore, Mike, et al. "Quantifying the value of architecture design decisions: lessons from the field." Proceedings of the 25th International Conference on Software Engineering. IEEE Computer Society, 2003.

4. Kazman, Rick, Jai Asundi, and Mark Klien. Making architecture design decisions: An economic approach. No. CMU/SEI-2002-TR-035. CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERING INST, 2002.