ecogenenergy.info Religion Agile Analytics Pdf

AGILE ANALYTICS PDF

Friday, February 14, 2020


“One remarkable aspect of Agile Analytics is the breadth of coverage—from product and backlog management to Agile project management techniques. Download as PDF, TXT or read online from Scribd. Flag for . Chapter 1 Introducing Agile Analytics Alpine-Style Systems Development What Is Agile Analytics?. Keeping Up with the Data. Data needs to be refreshed regularly for you and your users to have confidence in your analytics. Your platform should offer a familiar.


Agile Analytics Pdf

Author:HEIDI DRAFTON
Language:English, Spanish, French
Country:Mozambique
Genre:Religion
Pages:154
Published (Last):09.02.2016
ISBN:533-3-15281-945-4
ePub File Size:20.73 MB
PDF File Size:20.88 MB
Distribution:Free* [*Regsitration Required]
Downloads:21364
Uploaded by: TAISHA

Agile Analytics in Higher. Education. Academic Data Warehouse. Alliance Session: March 20, PM. Full Text: PDF Agile Analytics by Ken Collier, Published by ACM . in software engineering rigorous and agile approaches: 2nd of june consider agile strategies for the BI/DW development projects. Keywords: BI, Agile strategies, Scrum, Extreme programming, Data warehousing, Analytics.

This architecture advocates a data pull in contrast to the traditional data push approach, much like Kanban systems. What I like about Kens book can be summarized by three points: This is quite an accomplishment. Anyone participating in data-centric or business analytics projects will benefit from this superb book. I had established the program two years earlier and worked closely with many of the officers to grow the chapters and host events.

As the business driver of the project, I knew exactly what functionality the chapter Web sites needed. I had researched registration and collaboration systems and mapped their capabilities to my feature matrix.

I was ready to wheel and deal and get a new system up and running in three months. Unfortunately, the project went corporate. The president assigned someone to manage the project, an IT person to collect requirements, and a marketing person to coordinate integration with our existing Web site. We established a regular time to meet and discuss solutions. In short order, the project died.

My first sense of impending doom came when I read the requirements document compiled by the IT developer after I had e-mailed her my requirements and had a short conversation.

When I read the documentand Im technically astuteI no longer recognized my project. I knew that anyone working from the document i. This experience made me realize how frustrated business people get with ITs traditional approach to software development. Because I witnessed how IT translates business requirements into IT-speak, I now had a greater understanding of why so many business intelligence BI projects fail. Agile to the rescue. When I first read about Agile development techniques, I rejoiced.

Someone with a tad of business and common sense had finally infiltrated the IT community. Everything about the methodology made perfect sense. Most important, it shifts the power in a development project from the IT team to business users for whom the solution is being built! However, the Agile development methodology was conceived to facilitate software projects for classic transaction-processing applications. Unfortunately, it didnt anticipate architecture- and data-laden development projects germane to business intelligence.

Fortunately, BI practitioners like Ken Collier have pioneered new territory by applying Agile methods to BI and have lived to tell about their experiences. Kens book is a fount of practical knowledge gleaned from real project work that shows the dos and donts of applying Agile methods to BI. Although the book contains a wealth of process knowledge, its not a howto manual; its really more of a rich narrative that gives would-be Agile BI practitioners the look, feel, smell, and taste of what its like to apply Agile methods in a real-world BI environment.

After you finish reading the book, you will feel as if you have worked side by side with Ken on a project and learned from the master. You may even have experienced the pain of a failed or failing project. Several years ago I worked for a midsize company that was seeking to replace its existing homegrown reporting application with a properly architected data warehouse. My role on the project was chief architect and technical lead.

This project ended very badly and our solution was ultimately abandoned. At the outset the project appeared poised for success and user satisfaction. However, in spite of the best efforts of developers, project managers, and stakeholders, the project ran over budget and over schedule, and the users were less than thrilled with the outcome.

It may have some similarities to projects that youve worked on. About the Project This section summarizes the essential characteristics of the project, including the following: Existing application. The companys existing reporting application was internally referred to as a data warehouse, which significantly skewed users understanding of what a data warehouse application offers.

In reality the data model was a replication of parts of one of the legacy operational databases. This replicated database did not include any data scrubbing and was wrapped in a significant amount of custom Java code to produce the reports required. Users had, at various times, requested new custom reports, and the application had become overburdened with highly specialized and seldom used reporting features.

All of the reports could be classified as canned reports. The system was not optimized for analytical activities, and advanced analytical capabilities were not provided. Project motivation. Because the existing data warehouse was not architected according to data warehousing best practices, it had reached the practical limits of maintainability and scalability needed to continue meeting user requirements.

Additionally, a new billing system was coming online, and it was evident that the existing system could not easily be adapted to accommodate the new data. Therefore, there was strong executive support for a properly designed data warehouse. External drivers. The data warehousing project was initially envisioned by a sales team from one of the leading worldwide vendors of data warehousing and business intelligence software. In providing guidance and presales support, this sales team helped the project sponsors understand the value of eliciting the help of experienced business intelligence consultants with knowledge of industry best practices.

However, as happens with many sales efforts, initial estimates of project scope, cost, and schedule were overly ambitious. Development team. The development team consisted exclusively of external data warehousing contractors. Because the companys existing IT staff had other high-priority responsibilities, there were no developers with deep knowledge of the business or existing operational systems. However, the development team had open access to both business and technical experts within the company as well as technology experts from the software vendor.

While initial discovery efforts were challenging, there was strong participation from all stakeholders. The primary customer for the new data warehouse was the companys finance department, and the project was sponsored by the chief financial officer.

They had a relatively focused business goal of gaining more reliable access to revenue and profitability information. They also had a substantial volume of existing reports used in business analysis on a routine basis, offering a reasonable basis for requirements analysis.

Project management.

The IT group was simultaneously involved in two other large development projects, both of which had direct or indirect impact on the data warehouse scope. Hosted environment. Because of limited resources and infrastructure, the companys IT leadership had recently decided to partner with an application service provider ASP to provide hosting services for newly developed production systems.

The data warehouse. While not insurmountable, this geographic separation did have implications for the movement of large volumes of data since operational systems remained on the east coast, residing on the corporate IT infrastructure.

Project Outcome The original project plan called for an initial data warehouse launch within three months but had an overly ambitious scope for this release cycle. Project completion was a full eight months after project start, five months late! User acceptance testing did not go well. Users were already annoyed with project delays, and when they finally saw the promised features, there was a large gap between what they expected and what was delivered.

As is common with late projects, people were added to the development team during the effort to try to get back on track. As Fred Brooks says, Adding more people to a late project only makes it later Brooks Ultimately, project costs far exceeded the budget, users were unsatisfied, and the project was placed on hold until further planning could be done to justify continued development.

Retrospective So who was to blame? Users felt that the developers had missed the mark and didnt implement all of their requirements. Developers felt that the users expectations were not properly managed, and the project scope grew out of control. Project sponsors felt that the vendors overpromised and underdelivered. Vendors felt that internal politics and organizational issues were to blame. Finally, many of the organizations IT staff felt threatened by lack of ownership and secretly celebrated the failure.

The project degenerated into a series of meetings to review contracts and project documents to see who should be held responsible, and guess what? Everyone involved was partially to blame. In addition to the normal technical challenges of data warehouse development, the following were identified as root causes of project failure: The contract did not sufficiently balance scope, schedule, and resources.

Requirements were incomplete, vague, and open-ended. There were conflicting interpretations of the previously approved requirements and design documents. Developers put in long nights and weekends in chaotic attempts to respond to user changes and new demands. The technical team was afraid to publicize early warning signs of impending failure and continued trying to honor unrealistic commitments.

Developers did not fully understand the users requirements or expectations, and they did not manage requirements changes well. Users had significant misconceptions about the purpose of a data warehouse since existing knowledge was based on the previous reporting application which was not a good model of a warehouse.

Vendors made ambitious promises that the developers could not deliver on in the time available. The project manager did not manage user expectations.

IT staff withheld important information from developers. The ASP partner did not provide the level of connectivity and technical support the developers expected. A higher degree of interaction among developers, users, stakeholders, and internal IT experts would have ensured accurate understanding on the part of all participants.

Early and frequent working software, no matter how simplistic, would have greatly reduced the users misconceptions and increased the accuracy of their expectations. Greater emphasis on user collaboration would have helped to avoid conflicting interpretations of requirements. A project plan that focused on adapting to changes rather than meeting a set of frozen contractual requirements would have greatly improved user satisfaction with the end product.

In the end, and regardless of blame, the root cause of this and many other data warehousing project failures is the disconnect in understanding and expectations between developers and users. Unfortunately, by the time I met Jim it was too late.

However, since then Jim and I have become good friends, exchanging ideas over coffee on a mostly weekly basis. Well, mostly he shares good ideas and I do my best to absorb them. Thats what this book is about: This adaptation is nontrivial as there are some very significant and unique challenges that we face that mainstream software developers do not.

Each practice will be thoroughly explained and demonstrated in a working example, and I will show you how you might modify each practice to best fit the uniqueness of your situation. This book is written for three broad audiences: For these readers I provide the details of Agile technical and project management techniques as they relate to business intelligence and data-centric projects.

For these readers I elaborate upon the traits of business intelligence projects and systems that make them distinctly different from software development projects, and I show how to adapt Agile principles and practices to these unique characteristics.

IT and engineering management who have responsibility for and oversight of program portfolios, including data warehousing, business intelligence, and analytics projects. This audience may possess neither deep technical expertise in business intelligence nor expertise in Agile methods. For these readers I present an introduction to an approach that promises to increase the likelihood of successful projects and delighted customers.

Readers already familiar with business intelligence should feel free to skip over these sections. By the way, although Im not an expert in all types of enterprise IT systems, such as enterprise resource planning ERP implementations, I have reason to believe that the principles and practices that make up Agile Analytics can be easily adapted to work in those environments as well.

If you are an IT executive, you might consider the broader context of Agile development in your organization. In the last couple of years the Agile software development movement has exploded. Agile success stories abound. Empirical evidence continues to increase and strongly supports Agile software development. The Agile community has grown dramatically during the past few years, and many large companies have adopted agility across their IT and engineering departments.

And there has been a proliferation of books published about various aspects of Agile software development. Unfortunately, the popularity of Agile methods has been largely lost on the data and business intelligence communities. For some strange reason the data community and software development community have always tended to grow and evolve independently of one another.

Big breakthroughs that occur in one community are often lost on the other. The object-oriented boom of the s is a classic example of this. The software development community has reaped the tremendous benefits of folding object orientation into its DNA, yet object-oriented database development remains peripheral to the mainstream for the data community. Their arguments are wide and varied, and they are almost always based on myths, fallacies, and misunderstandings, such as It is too costly to evolve and change a data model.

You must complete the physical data model before you can begin developing reports and other user features. The reality is that there is nothing special about data-centric systems that makes Agile principles irrelevant or inappropriate.

The challenge is that Agile practices must be adapted, and a different tool set must be adopted for data-centric systems development. Although many of the current books on Agile concepts and techniques are directly relevant to the data community, most of them do not speak directly to the data-minded reader. Unfortunately, many current Agile books are too narrowly focused on new, greenfield software development using all the latest platforms, frameworks, and programming languages.

It can be difficult for readers to extrapolate the ideas presented in these books to database development, data warehouse development, ERP implementation, legacy systems development, and so forth.

Agile author and database expert Scott Ambler has written books on Agile database development and database refactoring a distinctly Agile practice to engage the database community in the Agile dialogue. Ralph does a great job of adapting Scrum and eXtreme Programming XP techniques to the nuances of data warehousing, and many of those concepts are also present in this book. Additionally, this book aims to dive into many of the technical practices that are needed to develop in an Agile manner.

A word about terminology: Ive chosen the title Agile Analytics more because its catchy and manageable than because it precisely captures my focus. By and large the data warehousing community has come to use the term data warehousing to refer to back-end management and preparation of data for analysis and business intelligence to refer to the user-facing front-end applications that present data from the warehouse for analysis.

The term analytics is frequently used to suggest more advanced business intelligence methods involving quantitative analysis of data e. Moreover, the industry term business intelligence is sometimes an ambiguous and broadly encompassing term that includes anything to do with data-driven business processes business performance management, customer relationship management, etc. My use of the moniker Agile Analytics should not imply that Agile methods are applicable only to a certain class of user-facing BI application development.

Agile methods are applicable and adaptable to data warehouse development as well as business intelligence and analytical application development. For many people Agile BI development tends to be easier to imagine, since it is often assumed that the data warehouse has been built and populated.

Certainly a preexisting data warehouse simplifies the effort required to build BI applications. However, you should not take this to mean that the data warehouse must be completed prior to building BI applications.

In fact, Agile Analytics is a user-valuedriven approach in which high-valued BI capabilities drive the evolutionary development of the data warehouse components needed to support those capabilities. In this way we avoid overbuilding the warehouse to support more than its intended purpose. My use of the term business intelligence or BI throughout this book should be assumed to include analytic as well as reporting and querying applications.

In most of these cases you should assume that I mean to include relevant DW components as well. Ill also address some of the advanced BI concepts like data mining. Ill leave it to the reader to extrapolate the practices to more specific BI projects such as CRM implementations. The principles still apply. These communities are connected by a project manager, a business analyst or product owner , and an executive sponsor. Each of these communities plays a crucial role in project success, and each of these communities requires a well-defined set of Agile practices to be effective in its role.

This book is intended for both business and technical readers who are involved in one or more of the communities described. Not everything in the book is meant for everyone on the list, but there is something here for everyone. I have worked with many organizations that seek Agile training, mentoring, and coaching. Occasionally I have to dispel the myth that agility applies only to developers and techies. At one company with which I was invited to work, the executive who sponsored the training said something like, If our engineers could just start doing Agile development, we could finish projects faster and our customers would be happier.

This statement represents some unfortunate misconceptions that can be a buzzkill for Agile teams. First, successful agility requires a change in the mind-set of all team members. Customer community members must understand that their time is required to explore and exercise newly completed features, and to provide continuous input and feedback on the same.

Management community members must adapt their expectations as project risk and uncertainty unfolds, and as the team adapts to inevitable change. The technical community must learn a whole new way of working that involves lots of discipline and rigor. And the project interface community must be committed to daily project involvement and a shift in their role and contribution to project success. Second, Agile doesnt always mean faster project completion.

Even the best project teams still have a finite capacity to complete a scope of work. Agility is not a magic wand that makes teams work faster. Agile practices do steer. However, I would caution against expecting significantly faster project cycles, especially in the beginning. For this reason I have tried to design this book to provide something relevant for everyone.

Part I, Agile Analytics: Management Methods, is focused on Agile project management techniques and delivery team coordination. It includes the following chapters: Chapter 2, Agile Project Management, introduces an effective collection of practices for chartering, planning, executing, and monitoring an Agile Analytics project. Chapter 3, Community, Customers, and Collaboration, introduces a set of guidelines and practices for establishing a highly collaborative project community.

Chapter 4, User Stories for BI Systems, introduces the story-driven alternative to traditional requirements analysis and shows how use cases and user stories drive the continuous delivery of value. Chapter 5, Self-Organizing Teams Boost Performance, introduces an Agile style of team management and leadership as an effective alternative to more traditional command-and-control styles.

This first part is written for everyone involved in an Agile Analytics project, from executive sponsors, to project managers, to business analysts and product owners, to technical leads and delivery team members. These chapters establish a collection of core practices that shape the way an Agile project community works together toward a successful conclusion.

Part II of the book, Agile Analytics: Technical Methods, is focused on the technical methods that are necessary to enable continuous delivery of. This part includes the following chapters: Chapter 6, Evolving Excellent Design, shows how the evolutionary design process works and how to ensure that it results in higherquality data models and system components with minimal technical debt.

Chapter 7, Test-Driven Data Warehouse Development, introduces a collection of practices and tools for automated testing, and for taking a test-first approach to building data warehouse and business intelligence components. Chapter 9, Project Automation, shows how to combine test automation and version control practices to establish an automated continuous integration environment that maintains confidence in the quality of the evolving system. Chapter 10, Final Words, takes a look at some of the remaining factors and considerations that are critical to the successful adoption of an Agile Analytics approach.

However, these technical practices are essential when an Agile Analytics approach is taken. Of course, these technical chapters should be read by technical team leads and delivery team members. However, I also recommend that nontechnical project team members read the introductory sections of each of these chapters. Doing so will help nontechnical members establish a shared understanding of the purpose of these practices and appreciate the value of the technical teams efforts to apply them.

I like to think of Agile Analytics techniques as supporting one of the following focal points:. Each chapter is dedicated to a key practice or related set of practices, beginning with an executive-level overview of the salient points of the chapter and progressing into deeper coverage of the topic.

Some of the chapter topics are rich enough to deserve to be entire books. In these cases, my aim is to give the reader a solid understanding of the topic, and ideally the motivation needed for a deeper self-study of its mechanics.

If you are a data warehouse manager, project sponsor, or anyone who needs to have a good working understanding of the practices without getting bogged down in the technical details, I recommend reading the middle sections of each chapter, especially the project management chapters. These sections are designed to provide a deep enough understanding of the topic to either use the techniques or understand how they are used on your project.

If you are a member of the day-to-day project team project managers, technical team members, business analysts, product managers, etc. Management Methods.

Log in to Wiley Online Library

These are designed to give you a concrete set of techniques to apply in your release planning, iteration planning, and all other project management and user collaboration activities. Technical Methods, are intended for you. I am a technology agnostic. As this book goes to press, there are an increasing number of data warehouse and business intelligence tool vendors that are branding their products as Agile. Tools and tool suites from forward-thinking vendors such as WhereScape, Pentaho, Balanced Insight, and others offer some exciting possibilities for enabling agility.

While I do not believe that you must have these types of tools to take an Agile approach, they certainly do offer some powerful benefits to Agile delivery teams. The Agile software development community has greatly benefited from tools that help automate difficult development activities, and I look forward to the benefits that our community stands to gain from these vendors.

At the same time I would caution you not to believe that you must have such tools before you can start being Agile. Instead, I encourage you to get started with Agile techniques and practices and adopt tools incrementally as you determine that they are of sufficient benefit.

These friends and colleagues have my respect and gratitude for the many valuable interactions Ive had with them, and the collaborations that ultimately resulted in the Agile Analytics approach. Foremost, my good friend Jim Highsmith has been my trusted adviser and mentor since the beginning of my Agile journey. Jim was just starting to write the first edition of Agile Project Management when I first met him, and he made book-writing look so easy that I decided to give it a try.

As it turns out, its much harder than he makes it look. My weekly breakfast discussions with Jim were critical in shaping the concepts in this book. He voluntarily served as my developmental editor, reviewing early drafts of sections and chapters and helping me pull things together in a more cohesive and coherent fashion.

Jim continues to challenge my assumptions and gives me new ideas and new ways to think about the complexities of development. He also didnt give up on me when book-writing wasnt my highest priority. Thanks, Jim. Jim introduced me to Luke Hohmann at a time when Luke was looking for somebody with both data warehousing experience and Agile knowledge. Luke is one of the most visionary people Ive ever met. I was fortunate enough to be the chief architect for one of Lukes innovative ideas: The concepts in this book stem from that experience and have been refined and matured on subsequent projects.

Luke has become a great friend over the past seven years, and I value his wisdom and vision. Thanks, Luke. My team on the aforementioned project remains one of the best Agile teams I have yet experienced either as a participant or as an Agile trainer. Each team member brought a special set of skills and perspectives, and over that first three-plus-year-long project these friends and teammates helped me figure xxxiii.

Ive since had other project opportunities to work with many of these friends, further refining Agile Analytics concepts.

These team members deserve much of the credit for validating and tweaking Agile Analytics practices in a complex and real-life situation. Thanks, guys. Jim Highsmith also introduced me to Scott Ambler along the way. Scott has led the charge in applying Agile to data-centric systems development. Fortunately for all of us, Scott is a prolific writer who freely shares his ideas in his many books and on his ambysoft.

Scott gave much of his time reviewing this book. He has given me much to think about and shared ideas with me that I might otherwise have missed. Thanks, Scott. I dont think I truly understood what it means for somebody to have the patience of a saint before working with Addison-Wesley editor Chris Guzikowski and editorial assistant Raina Chrobak.

As it turns out, I am a painfully slow author who is not very good at applying Agile principles to book-writing deadlines. Huge thanks go to Raina and Chris, who were amazingly patient as I slipped deadline after deadline.

I hope I have future opportunities to redeem myself as an author. Ralph and I were acquainted at that time and since have become friends and colleagues. I am grateful for his work in this area and for the discussions Ive had with him and the experiences he has shared. Although I have tried not to duplicate what Ralph has already published, I am confident that our approaches are consistent with and complementary to one another. I look forward to future collaborations with Ralph as our ideas mature and evolve.

Finally, the ideas presented in this book have benefited tremendously from smart and thoughtful people willing to review its early drafts and give me guidance.

In addition to Scotts and Jims reviews, special thanks go to Jonathon Golden, my go-to guru on project automation, and Israel Gat, expert on Agile leadership and technical debt. Their contributions were invaluable. Ken has successfully adapted Agile techniques to data warehousing and business intelligence to create the Agile Analytics style. Ken also frequently trains data warehousing and business intelligence teams in Agile Analytics, giving him the opportunity to exercise this approach with various technologies, team dynamics, and industry domains.

In nearly three decades of working in advanced computing and technology, Ken has experienced many of the trends that come and go in our field, as well as the ones that truly transform the state of our practices. With an M. Ken also loves helping organizations adopt and tailor effective approaches and solutions that might not otherwise be apparent. He combines a deep technical foundation with sound business acumen to help bridge the gaps that often exist between technical and business professionals.

Ken has had the privilege of working as a software engineer for a large semiconductor company. He has spent several years as a tenured professor of computer science engineering. He has directed the data warehousing and business intelligence solutions group for a major consulting firm. And, most recently, he has focused on enabling organizational agility, including Agile software engineering, Agile Analytics, and Agile management and leadership for client companies.

It is not a rigid or prescriptive methodology; rather it is a style of building a data warehouse, data marts, business intelligence applications, and analytics applications that focuses on the early and continuous delivery of business value throughout the development lifecycle.

Agile Analytics includes practices for project planning, management, and monitoring; for effective collaboration with your business customers and management stakeholders; and for ensuring technical excellence by the delivery team. This chapter outlines the tenets of Agile Analytics and establishes the foundational principles behind each of the practices and techniques that are introduced in the successive chapters in this book. Agile is a reserved word when used to describe a development style.

It means something very specific. Unfortunately, agile occasionally gets misused as a moniker for processes that are ad hoc, slipshod, and lacking in discipline. Agile relies on discipline and rigor; however, it is not a heavyweight or highly ceremonious process despite the attempts of some methodologists to codify it with those trappings.

Rather, Agile falls somewhere in the middle between just enough structure and just enough flexibility. It has been said that Agile is simple but not easy, describing the fact that it is built on a simple set of sensible values and principles but requires a high degree of discipline and rigor to properly execute.

It is important to accurately understand the minimum set of characteristics that differentiate a true Agile process from those that are too unstructured or too rigid. This chapter is intended to leave you with a clear understanding of those characteristics as well as the underlying values and principles of Agile Analytics.

These are derived directly from the tried and proven foundations established by the Agile software community and are adapted to the nuances of data warehousing and business intelligence development. Im fascinated by the trials and travails of climbing high mountains like Everest, Annapurna, and others that rise to over 8, meters above sea level. These expeditions are complicated affairs involving challenging planning and logistics, a high degree of risk and uncertainty, a high probability of death for every two climbers who reach the top of Annapurna, another one dies trying!

While it may not be as adventuresome, building complex business intelligence systems is a lot like high-altitude climbing. We face lots of risk and uncertainty, complex planning, difficult decisions in the heat of battle, and the likelihood of death! Okay, maybe not that last part, but you get the analogy.

Climbing teams first began successfully conquering these high mountains in the s, 60s, and 70s. In those early days the preferred mountaineering style was known as siege climbing, which had a lot of similarities to a military excursion. Expeditions were led in an autocratic command-andcontrol fashion, often by someone with more military leadership experience than climbing experience. Climbing teams were supported by the large numbers of porters required to carry massive amounts of gear and supplies to base camp and higher.

Mounting a siege-style expedition takes over a year of planning and can take two months or more to execute during the climbing season. Siege climbing is a yo-yo-like affair in which ropes are fixed higher and higher on the mountain, multiple semipermanent camps are established at various points along the route, and loads of supplies are relayed by porters to those higher camps.

Finally, with all this support, a small team of summit climbers launches the final push for the summit on a single day, leaving from the high camp and returning to the same. Brilliant teams have successfully climbed mountains for years in this style, but the expeditions are prohibitively expensive, time-consuming to execute, and fraught with heavyweight procedures and bureaucracy. Traditional business intelligence systems development is a lot like siege climbing.

It can result in high-quality, working systems that deliver the desired capabilities. However, these projects are typically expensive, exhibiting a lot of planning, extensive design prior to development, and long development cycles.

Like siege-style expeditions, all of the energy goes into one shot at the summit. If the summit bid fails, it is too time-consuming to return to base camp and regroup for another attempt. In my lifetime and Im not that old. When such projects fail, the typical management response is to cancel the project entirely rather than adjust, adapt, and regroup for another summit attempt.

In the s a new mountaineering method called alpine-style emerged, making it feasible for smaller teams to summit these high peaks faster, more cheaply, and with less protocol. Alpine-style mountaineering still requires substantial planning, a sufficient supporting team, and enough gear and supplies to safely reach the summit. However, instead of spending months preparing the route for the final summit push, alpine-style climbers spend about a week moving the bare essentials up to the higher camps.

In this style, if conditions are right, summits can be reached in a mere ten days. Teams of two to three climbers share a single tent and sleeping bag, fewer ropes are needed, and the climbers can travel much lighter and faster. When conditions are not right, it is feasible for alpine-style mountaineers to return to base camp and wait for conditions to improve to make another summit bid. It is essential that we have a sufficient amount of planning, the necessary support to be successful, and an appropriate amount of protocol.

Our summit is the completion of a high-quality, working business intelligence system that is of high value to its users. As in mountaineering, reaching our summit requires the proper conditions. We need just the right amount of planningbut we must be able to adapt our plan to changing factors and new information. We must prepare for a high degree of risk and uncertaintybut we must be able to nimbly manage and respond as risks unfold.

We need support and involvement from a larger communitybut we seek team self-organization rather than command-and-control leadership. Agile Analytics is a development style rather than a methodology or even a framework. The line between siege-style and alpine-style mountaineering is not precisely defined, and alpine-style expeditions may include some siege-style practices. Each style is best described in terms of its values and guiding principles. Each alpine-style expedition employs a distinct set of climbing practices that support a common set of values and principles.

Im not the first Agile advocate to discuss the analogy between climbing and Agile development. Jim Highsmith made a similar analogy in his book, Adaptive Software Development: Premier mountaineer Ed Viesturs has a formula, or core value, that is his cardinal rule in the big mountains: Getting to the top is optional. Getting down is mandatory. Viesturs and Roberts I love this core value because it is simple and elegant, and it provides a clear basis for all of Eds decision making when he is on the mountain.

In the stress of the climb, or in the midst of an intensely challenging project, we need just such a basis for decision makingour North Star. In , a group of the most influential application software developers convened in Salt Lake City and formed the Agile Alliance.

Through the process of sharing and comparing each of their styles of software development, the Agile Manifesto emerged as a simple and elegant basis for project guidance and decision making. The Agile Manifesto reads: Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.

With due respect to the Agile Alliance, of which I am a member, I have adapted the Agile Manifesto just a bit in order to make it more appropriate to Agile Analytics: Manifesto for Agile Analytics Development We are uncovering better ways of developing data warehousing and business intelligence systems by doing it and helping others do it.

In addition to dealing with large volumes of data, our efforts involve systems integration, customization, and programming. Every activity related to any project either a directly and materially contributes to this primary objective or b does not. Agile Analytics attempts to maximize a-type activities while acknowledging that there are some b-type activities that are still important, such as documenting your enterprise data model. These include technical, project management, and user collaboration practices.

I will demonstrate how you can apply these on your projects, and how you can tailor them to the nuances of your environment. However, the title of this section is What Is Agile Analytics? This is simply a high-level glimpse at the key project traits that are the mark of agility, not an exhaustive list of practices.

Moreover, Agile Analytics is a development style, not a prescriptive methodology that tells you precisely what you must do and how you must do it. The dynamics of each project within each organization require practices that can be tailored appropriately to the environment.

These characteristics simply serve that goal: Iterative, incremental, evolutionary. Foremost, Agile is an iterative, incremental, and evolutionary style of development. We work in short iterations that are generally one to three weeks long, and never more than four weeks. We build the system in small increments or chunks of user-valued functionality. And we evolve the working system by adapting to frequent user feedback. Agile development is like driving around in an unfamiliar city; you want to avoid going very far without some validation that you are on the right course.

Short iterations with frequent user reviews help ensure that we are never very far off course in our development. Value-driven development. The goal of each development iteration is the production of user-valued features.

While you and I may appreciate the difficulty of complex data architectures, elegant data models, efficient ETL scripts, and so forth, users generally couldnt care less about these things. Production quality. Each newly developed feature must be fully tested and debugged during the development iteration. Agile development is not about building hollow prototypes; it is about incrementally evolving to the right solution with the best architectural underpinnings.

A user feature is Done when it is of production quality, it is successfully integrated into the evolving system, and developers are proud of their work. That same feature is Done! Barely sufficient processes. Ive worked on many projects that involved elaborate stage-gate meetings between stages of development such as the transition from requirements analysis to design.

These gates are almost always accompanied by a formal document that must be signed off as part of the gating process. If a data dictionary is deemed important for use by future developers, then perhaps a digital image of a whiteboard table or a simple spreadsheet table will suffice.

Since our primary objective is the production of high-quality, high-value, working systems, we must be able to minimize the amount of ceremony required for other activities. Historically database and data warehouse testing has lacked the rigor, discipline, and automation that have benefited software development efforts www.

Automation, automation, automation.

The only way to be truly Agile is to automate as many routine processes as possible. Test automation is perhaps the most critical. If you must test your features and system manually, guess how often youre likely to rerun your tests?

Test automation enables you to frequently revalidate that everything is still working as expected. This helps establish continuous confidence that you are never more than a few hours or days away from putting a new version into production. Agile Analytics teams seek to automate any process that is done more than once. The more you can automate, the more you can focus on developing user features.

Too often in traditional projects the development team solely bears the burden of ensuring that timelines are met, complete scope is delivered, budgets are managed, and quality is ensured. Agile business intelligence acknowledges that there is a broader project community that shares responsibility for project success. The project community includes the subcommunities of users, business owners, stakeholders, executive sponsors, technical experts, project managers, and others.

Frequent collaboration between the technical and user communities is critical to success. Daily collaboration within the technical community is also critical. In fact, establishing a collaborative team workspace is an essential ingredient of successful Agile projects.

Self-organizing, self-managing teams. Hire the best people, give them the tools and support they need, then stand aside and allow them to be successful. There is a key shift in the Agile project management style compared to traditional project management. The Agile project managers role is to enable team members to work their magic and to facilitate a high degree of collaboration with users and other members of the project community.

The Agile project team decides how much work it can complete during an iteration, then holds itself accountable to honor those commitments. The Agile style is not a substitute for having the right people on the team.

These principles often become the tiebreaker when difficult trade-off decisions must be made. Similarly, the Agile Alliance has established a set of principles for. We welcome changing requirements, even late in development. Users, stakeholders, and developers must share project ownership and work together daily throughout the project. We value the importance of talented and experienced business intelligence experts. We give them the environment and support they need and trust them to get the job done.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. A working business intelligence system is the primary measure of progress. We recognize the balance among project scope, schedule, and cost. The data warehousing team must work at a sustainable pace.

Continuous attention to the best data warehousing practices enhances agility. The best architectures, requirements, and designs emerge from selforganizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. Take a minute to reflect on these principles.

How many of them are present in the projects in your organization? Do they make sense for your organization? Give them another look. Are they realistic principles for your organization? I have found these not only to be commonsense principles, but also to be effective and achievable on real projects.

Furthermore, adherence to these principles rather than reliance on a prescriptive and ceremonious process model is very liberating. Agile Analytics is not: A wholesale replacement of traditional practices. It is important that we keep the practices and methods that work well, improve those that allow room for improvement, and replace those that are problematic. Scrum is perhaps the Agile flavor that has received the most publicity along with XP in recent years. However, it is incorrect to say that Agile was formerly known as eXtreme Programming, as one skeptic told me.

In fact, there are many different Agile development flavors that add valuable principles and practices to the broader collective known as Agile development. Agile Analytics is an adaptation of principles and practices from a variety of these methods to the complexities of data-intensive, analytics-based systems integration efforts like data warehousing and data mart development.

Simply iterating. Short, frequent development iterations are an essential cornerstone of Agile development. Unfortunately, this key practice is commonly misconstrued as the definition of agility. Not long ago I was asked to mentor a development team that had gone Agile but wasnt experiencing the expected benefits of agility.

Upon closer inspection I discovered that they were planning in four-week iterations but didnt expect to have any working features until 5. Effectively they had divided the traditional waterfall model into time blocks they called iterations. They completely missed the point.

The aim of iterative development is to demonstrate working features and to obtain frequent feedback from the user community. This means that every iteration must result in demonstrable working software. For systems integration; its only for programming. This complexity calls for an approach that supports a high degree of customer collaboration, frequent delivery of working software, and frequent feedbackaha, an Agile approach!

An excuse for ad hoc behavior. Some have mistaken the tenets of Agile development for abandonment of rigor, quality, or structure, in other words, hacking.

This misperception could not be farther from the truth. Agility is a focus on the frequent delivery of highvalue, production-quality, working software to the user community with the goal of continuously adapting to user feedback.

This means that automated testing and quality assurance are critical components of all iterative development activities. We dont build prototypes; we build working features and then mature those features in response to user input. Others mistake the Agile Manifesto as disdain of documentation, which is also incorrect. The keyword here is sufficient. Sufficiency implies that there is a legitimate purpose for the document, and when that purpose is served, there is no need for additional documentation.

This is a natural inclination for new Agile practitioners, and I will provide some. However, I need to reemphasize that Agile Analytics is a style, not a methodology and not a framework. Figuratively, you can absorb agility into your DNA with enough focus, practice, and discipline. Youll know youve reached that point when you begin applying Agile principles to everything you do such as buying a new car, remodeling a bathroom, or writing a book.

This is not a substitute for any of the more comprehensive technical books on data warehousing but should be sufficient as a baseline for the remainder of the book. Data Warehousing Conceptual Architectures Figure 1.

This is a high-level conceptual architecture containing multiple layers, each of which includes a complex integration of commercial technologies, data modeling and manipulation, and some custom code.

The data warehouse architecture includes one or more operational source systems from which data is extracted, transformed, and loaded into the data Source Systems Intergration Presentation Analysis. These systems are optimized for the daily transactional processing required to run the business operations. Data from these sources is extracted into an integration tier in the architecture that acts as a holding pen where data can be merged, manipulated, transformed, cleansed, and validated without placing an undue burden on the operational systems.

This tier may include an operational data store or an enterprise information integration EII repository that acts as a system of record for all relevant operational data. The integration database is typically based on a relational data model and may have multiple subcomponents, including pre-staging, staging, and an integration repository, each serving a different purpose relating to the consolidation and preprocessing of data from disparate source systems. Data is extracted from the staging database, transformed, and loaded into a presentation tier in the architecture that contains appropriate structures for optimized multidimensional and analytical queries.

This system is designed to support the data slicing and dicing that define the power of a data warehouse. There are a variety of alternatives for the implementation of the presentation database, including normalized relational schemas and denormalized schemas like star, snowflake, and even starflake. Moreover, the presentation tier may include a single enterprise data warehouse or a collection of subject-specific data marts.

Some architectures include a hybrid of both of these. Presentation repositories are typically implemented in the same technologies as the integration database. Finally, data is presented to the business users at the analysis tier in the architecture. This conceptual layer in the system represents the variety of applications and tools that provide users with access to the data, including report writers, ad hoc querying, online analytical processing OLAP , data visualization, data mining, and statistical analysis.

BI tool vendors such as Pentaho, Cognos, MicroStrategy, Business Objects, Microsoft, Oracle, IBM, and others produce commercial products that enable data from the presentation database to be aggregated and presented within user applications. This is a generalized architecture, and actual implementations vary in the details. One major variation on the Kimball architecture is the Inmon architecture Inmon , which inserts a layer of subject-specific data marts that contain subsets of the data from the main warehouse.

Each data mart supports only the specific end-user applications that are relevant to the business subject area for which that mart was designed. Regardless of your preferences for Kimball- versus Inmon-style architectures, and of the variations found in implementation detail, Figure 1.

Diverse and Disparate Technical Skills Inherent in the implementation of this architecture are the following aspects of development, each requiring a unique set of development skills: While he advocates architecture evolution and evolutionary design is covered in Chapter 6, Evolving Excellent Design , he describes architectural patterns that are adaptive.

In Chapter 6 he introduces an adaptable analytics architecture, one that he used on a large project in which change over time was a key part of the challenge. This architecture advocates a data pull in contrast to the traditional data push approach, much like Kanban systems.

What I like about Kens book can be summarized by three points: This is quite an accomplishment. Anyone participating in data-centric or business analytics projects will benefit from this superb book. I had established the program two years earlier and worked closely with many of the officers to grow the chapters and host events.

As the business driver of the project, I knew exactly what functionality the chapter Web sites needed.

I had researched registration and collaboration systems and mapped their capabilities to my feature matrix. I was ready to wheel and deal and get a new system up and running in three months.

Unfortunately, the project went corporate. The president assigned someone to manage the project, an IT person to collect requirements, and a marketing person to coordinate integration with our existing Web site. We established a regular time to meet and discuss solutions.

In short order, the project died.

My first sense of impending doom came when I read the requirements document compiled by the IT developer after I had e-mailed her my requirements and had a short conversation. When I read the documentand Im technically astuteI no longer recognized my project. I knew that anyone working from the document i. This experience made me realize how frustrated business people get with ITs traditional approach to software development.

Because I witnessed how IT translates business requirements into IT-speak, I now had a greater understanding of why so many business intelligence BI projects fail. Agile to the rescue. When I first read about Agile development techniques, I rejoiced.

Someone with a tad of business and common sense had finally infiltrated the IT community. Everything about the methodology made perfect sense.

Most important, it shifts the power in a development project from the IT team to business users for whom the solution is being built! However, the Agile development methodology was conceived to facilitate software projects for classic transaction-processing applications. Unfortunately, it didnt anticipate architecture- and data-laden development projects germane to business intelligence. Fortunately, BI practitioners like Ken Collier have pioneered new territory by applying Agile methods to BI and have lived to tell about their experiences.

Kens book is a fount of practical knowledge gleaned from real project work that shows the dos and donts of applying Agile methods to BI. Although the book contains a wealth of process knowledge, its not a howto manual; its really more of a rich narrative that gives would-be Agile BI practitioners the look, feel, smell, and taste of what its like to apply Agile methods in a real-world BI environment.

After you finish reading the book, you will feel as if you have worked side by side with Ken on a project and learned from the master. You may even have experienced the pain of a failed or failing project. Several years ago I worked for a midsize company that was seeking to replace its existing homegrown reporting application with a properly architected data warehouse.

My role on the project was chief architect and technical lead. This project ended very badly and our solution was ultimately abandoned. At the outset the project appeared poised for success and user satisfaction.

However, in spite of the best efforts of developers, project managers, and stakeholders, the project ran over budget and over schedule, and the users were less than thrilled with the outcome. It may have some similarities to projects that youve worked on.

About the Project This section summarizes the essential characteristics of the project, including the following: Existing application. The companys existing reporting application was internally referred to as a data warehouse, which significantly skewed users understanding of what a data warehouse application offers. In reality the data model was a replication of parts of one of the legacy operational databases.

This replicated database did not include any data scrubbing and was wrapped in a significant amount of custom Java code to produce the reports required. Users had, at various times, requested new custom reports, and the application had become overburdened with highly specialized and seldom used reporting features. All of the reports could be classified as canned reports.

The system was not optimized for analytical activities, and advanced analytical capabilities were not provided. Project motivation. Because the existing data warehouse was not architected according to data warehousing best practices, it had reached the practical limits of maintainability and scalability needed to continue meeting user requirements.

Additionally, a new billing system was coming online, and it was evident that the existing system could not easily be adapted to accommodate the new data. Therefore, there was strong executive support for a properly designed data warehouse. External drivers. The data warehousing project was initially envisioned by a sales team from one of the leading worldwide vendors of data warehousing and business intelligence software.

In providing guidance and presales support, this sales team helped the project sponsors understand the value of eliciting the help of experienced business intelligence consultants with knowledge of industry best practices. However, as happens with many sales efforts, initial estimates of project scope, cost, and schedule were overly ambitious. Development team. The development team consisted exclusively of external data warehousing contractors. Because the companys existing IT staff had other high-priority responsibilities, there were no developers with deep knowledge of the business or existing operational systems.

However, the development team had open access to both business and technical experts within the company as well as technology experts from the software vendor. While initial discovery efforts were challenging, there was strong participation from all stakeholders. The primary customer for the new data warehouse was the companys finance department, and the project was sponsored by the chief financial officer.

They had a relatively focused business goal of gaining more reliable access to revenue and profitability information. They also had a substantial volume of existing reports used in business analysis on a routine basis, offering a reasonable basis for requirements analysis.

Project management. The IT group was simultaneously involved in two other large development projects, both of which had direct or indirect impact on the data warehouse scope.

Hosted environment. Because of limited resources and infrastructure, the companys IT leadership had recently decided to partner with an application service provider ASP to provide hosting services for newly developed production systems. The data warehouse. While not insurmountable, this geographic separation did have implications for the movement of large volumes of data since operational systems remained on the east coast, residing on the corporate IT infrastructure.

Project Outcome The original project plan called for an initial data warehouse launch within three months but had an overly ambitious scope for this release cycle. Project completion was a full eight months after project start, five months late! User acceptance testing did not go well. Users were already annoyed with project delays, and when they finally saw the promised features, there was a large gap between what they expected and what was delivered.

As is common with late projects, people were added to the development team during the effort to try to get back on track. As Fred Brooks says, Adding more people to a late project only makes it later Brooks Ultimately, project costs far exceeded the budget, users were unsatisfied, and the project was placed on hold until further planning could be done to justify continued development.

Retrospective So who was to blame? Users felt that the developers had missed the mark and didnt implement all of their requirements. Developers felt that the users expectations were not properly managed, and the project scope grew out of control.

Project sponsors felt that the vendors overpromised and underdelivered. Vendors felt that internal politics and organizational issues were to blame. Finally, many of the organizations IT staff felt threatened by lack of ownership and secretly celebrated the failure. The project degenerated into a series of meetings to review contracts and project documents to see who should be held responsible, and guess what?

Everyone involved was partially to blame. In addition to the normal technical challenges of data warehouse development, the following were identified as root causes of project failure: The contract did not sufficiently balance scope, schedule, and resources. Requirements were incomplete, vague, and open-ended.

There were conflicting interpretations of the previously approved requirements and design documents. Developers put in long nights and weekends in chaotic attempts to respond to user changes and new demands. The technical team was afraid to publicize early warning signs of impending failure and continued trying to honor unrealistic commitments. Developers did not fully understand the users requirements or expectations, and they did not manage requirements changes well.

Users had significant misconceptions about the purpose of a data warehouse since existing knowledge was based on the previous reporting application which was not a good model of a warehouse. Vendors made ambitious promises that the developers could not deliver on in the time available. The project manager did not manage user expectations. IT staff withheld important information from developers. The ASP partner did not provide the level of connectivity and technical support the developers expected.

A higher degree of interaction among developers, users, stakeholders, and internal IT experts would have ensured accurate understanding on the part of all participants. Early and frequent working software, no matter how simplistic, would have greatly reduced the users misconceptions and increased the accuracy of their expectations.

Greater emphasis on user collaboration would have helped to avoid conflicting interpretations of requirements. A project plan that focused on adapting to changes rather than meeting a set of frozen contractual requirements would have greatly improved user satisfaction with the end product. In the end, and regardless of blame, the root cause of this and many other data warehousing project failures is the disconnect in understanding and expectations between developers and users.

Unfortunately, by the time I met Jim it was too late. However, since then Jim and I have become good friends, exchanging ideas over coffee on a mostly weekly basis. Well, mostly he shares good ideas and I do my best to absorb them. Thats what this book is about: This adaptation is nontrivial as there are some very significant and unique challenges that we face that mainstream software developers do not. Each practice will be thoroughly explained and demonstrated in a working example, and I will show you how you might modify each practice to best fit the uniqueness of your situation.

This book is written for three broad audiences: For these readers I provide the details of Agile technical and project management techniques as they relate to business intelligence and data-centric projects. For these readers I elaborate upon the traits of business intelligence projects and systems that make them distinctly different from software development projects, and I show how to adapt Agile principles and practices to these unique characteristics.

IT and engineering management who have responsibility for and oversight of program portfolios, including data warehousing, business intelligence, and analytics projects. This audience may possess neither deep technical expertise in business intelligence nor expertise in Agile methods.

For these readers I present an introduction to an approach that promises to increase the likelihood of successful projects and delighted customers.

Readers already familiar with business intelligence should feel free to skip over these sections. By the way, although Im not an expert in all types of enterprise IT systems, such as enterprise resource planning ERP implementations, I have reason to believe that the principles and practices that make up Agile Analytics can be easily adapted to work in those environments as well.

If you are an IT executive, you might consider the broader context of Agile development in your organization. In the last couple of years the Agile software development movement has exploded. Agile success stories abound. Empirical evidence continues to increase and strongly supports Agile software development. The Agile community has grown dramatically during the past few years, and many large companies have adopted agility across their IT and engineering departments.

And there has been a proliferation of books published about various aspects of Agile software development. Unfortunately, the popularity of Agile methods has been largely lost on the data and business intelligence communities. For some strange reason the data community and software development community have always tended to grow and evolve independently of one another. Big breakthroughs that occur in one community are often lost on the other. The object-oriented boom of the s is a classic example of this.

The software development community has reaped the tremendous benefits of folding object orientation into its DNA, yet object-oriented database development remains peripheral to the mainstream for the data community. Their arguments are wide and varied, and they are almost always based on myths, fallacies, and misunderstandings, such as It is too costly to evolve and change a data model.

You must complete the physical data model before you can begin developing reports and other user features. The reality is that there is nothing special about data-centric systems that makes Agile principles irrelevant or inappropriate. The challenge is that Agile practices must be adapted, and a different tool set must be adopted for data-centric systems development. Although many of the current books on Agile concepts and techniques are directly relevant to the data community, most of them do not speak directly to the data-minded reader.

Unfortunately, many current Agile books are too narrowly focused on new, greenfield software development using all the latest platforms, frameworks, and programming languages. It can be difficult for readers to extrapolate the ideas presented in these books to database development, data warehouse development, ERP implementation, legacy systems development, and so forth.

Agile author and database expert Scott Ambler has written books on Agile database development and database refactoring a distinctly Agile practice to engage the database community in the Agile dialogue. Ralph does a great job of adapting Scrum and eXtreme Programming XP techniques to the nuances of data warehousing, and many of those concepts are also present in this book. Additionally, this book aims to dive into many of the technical practices that are needed to develop in an Agile manner.

A word about terminology: Ive chosen the title Agile Analytics more because its catchy and manageable than because it precisely captures my focus. By and large the data warehousing community has come to use the term data warehousing to refer to back-end management and preparation of data for analysis and business intelligence to refer to the user-facing front-end applications that present data from the warehouse for analysis.

The term analytics is frequently used to suggest more advanced business intelligence methods involving quantitative analysis of data e. Moreover, the industry term business intelligence is sometimes an ambiguous and broadly encompassing term that includes anything to do with data-driven business processes business performance management, customer relationship management, etc.

My use of the moniker Agile Analytics should not imply that Agile methods are applicable only to a certain class of user-facing BI application development. Agile methods are applicable and adaptable to data warehouse development as well as business intelligence and analytical application development.

For many people Agile BI development tends to be easier to imagine, since it is often assumed that the data warehouse has been built and populated.

Certainly a preexisting data warehouse simplifies the effort required to build BI applications. However, you should not take this to mean that the data warehouse must be completed prior to building BI applications. In fact, Agile Analytics is a user-valuedriven approach in which high-valued BI capabilities drive the evolutionary development of the data warehouse components needed to support those capabilities. In this way we avoid overbuilding the warehouse to support more than its intended purpose.

My use of the term business intelligence or BI throughout this book should be assumed to include analytic as well as reporting and querying applications. In most of these cases you should assume that I mean to include relevant DW components as well. Ill also address some of the advanced BI concepts like data mining. Ill leave it to the reader to extrapolate the practices to more specific BI projects such as CRM implementations. The principles still apply. These communities are connected by a project manager, a business analyst or product owner , and an executive sponsor.

Each of these communities plays a crucial role in project success, and each of these communities requires a well-defined set of Agile practices to be effective in its role. This book is intended for both business and technical readers who are involved in one or more of the communities described.

Not everything in the book is meant for everyone on the list, but there is something here for everyone. I have worked with many organizations that seek Agile training, mentoring, and coaching. Occasionally I have to dispel the myth that agility applies only to developers and techies. At one company with which I was invited to work, the executive who sponsored the training said something like, If our engineers could just start doing Agile development, we could finish projects faster and our customers would be happier.

This statement represents some unfortunate misconceptions that can be a buzzkill for Agile teams. First, successful agility requires a change in the mind-set of all team members. Customer community members must understand that their time is required to explore and exercise newly completed features, and to provide continuous input and feedback on the same.

Management community members must adapt their expectations as project risk and uncertainty unfolds, and as the team adapts to inevitable change.

The technical community must learn a whole new way of working that involves lots of discipline and rigor. And the project interface community must be committed to daily project involvement and a shift in their role and contribution to project success. Second, Agile doesnt always mean faster project completion.

Even the best project teams still have a finite capacity to complete a scope of work. Agility is not a magic wand that makes teams work faster.

Agile practices do steer. However, I would caution against expecting significantly faster project cycles, especially in the beginning. For this reason I have tried to design this book to provide something relevant for everyone.

Part I, Agile Analytics: Management Methods, is focused on Agile project management techniques and delivery team coordination. It includes the following chapters: Chapter 2, Agile Project Management, introduces an effective collection of practices for chartering, planning, executing, and monitoring an Agile Analytics project. Chapter 3, Community, Customers, and Collaboration, introduces a set of guidelines and practices for establishing a highly collaborative project community.

Chapter 4, User Stories for BI Systems, introduces the story-driven alternative to traditional requirements analysis and shows how use cases and user stories drive the continuous delivery of value.

Chapter 5, Self-Organizing Teams Boost Performance, introduces an Agile style of team management and leadership as an effective alternative to more traditional command-and-control styles. This first part is written for everyone involved in an Agile Analytics project, from executive sponsors, to project managers, to business analysts and product owners, to technical leads and delivery team members.

These chapters establish a collection of core practices that shape the way an Agile project community works together toward a successful conclusion. Part II of the book, Agile Analytics: Technical Methods, is focused on the technical methods that are necessary to enable continuous delivery of.

This part includes the following chapters: Chapter 6, Evolving Excellent Design, shows how the evolutionary design process works and how to ensure that it results in higherquality data models and system components with minimal technical debt.

Chapter 7, Test-Driven Data Warehouse Development, introduces a collection of practices and tools for automated testing, and for taking a test-first approach to building data warehouse and business intelligence components.

Chapter 9, Project Automation, shows how to combine test automation and version control practices to establish an automated continuous integration environment that maintains confidence in the quality of the evolving system. Chapter 10, Final Words, takes a look at some of the remaining factors and considerations that are critical to the successful adoption of an Agile Analytics approach.

However, these technical practices are essential when an Agile Analytics approach is taken. Of course, these technical chapters should be read by technical team leads and delivery team members.

However, I also recommend that nontechnical project team members read the introductory sections of each of these chapters. Doing so will help nontechnical members establish a shared understanding of the purpose of these practices and appreciate the value of the technical teams efforts to apply them. I like to think of Agile Analytics techniques as supporting one of the following focal points:.

Each chapter is dedicated to a key practice or related set of practices, beginning with an executive-level overview of the salient points of the chapter and progressing into deeper coverage of the topic. Some of the chapter topics are rich enough to deserve to be entire books. In these cases, my aim is to give the reader a solid understanding of the topic, and ideally the motivation needed for a deeper self-study of its mechanics.

If you are a data warehouse manager, project sponsor, or anyone who needs to have a good working understanding of the practices without getting bogged down in the technical details, I recommend reading the middle sections of each chapter, especially the project management chapters. These sections are designed to provide a deep enough understanding of the topic to either use the techniques or understand how they are used on your project.

If you are a member of the day-to-day project team project managers, technical team members, business analysts, product managers, etc. Management Methods. These are designed to give you a concrete set of techniques to apply in your release planning, iteration planning, and all other project management and user collaboration activities. Technical Methods, are intended for you.

I am a technology agnostic. As this book goes to press, there are an increasing number of data warehouse and business intelligence tool vendors that are branding their products as Agile. Tools and tool suites from forward-thinking vendors such as WhereScape, Pentaho, Balanced Insight, and others offer some exciting possibilities for enabling agility. While I do not believe that you must have these types of tools to take an Agile approach, they certainly do offer some powerful benefits to Agile delivery teams.

The Agile software development community has greatly benefited from tools that help automate difficult development activities, and I look forward to the benefits that our community stands to gain from these vendors. At the same time I would caution you not to believe that you must have such tools before you can start being Agile. Instead, I encourage you to get started with Agile techniques and practices and adopt tools incrementally as you determine that they are of sufficient benefit.

These friends and colleagues have my respect and gratitude for the many valuable interactions Ive had with them, and the collaborations that ultimately resulted in the Agile Analytics approach. Foremost, my good friend Jim Highsmith has been my trusted adviser and mentor since the beginning of my Agile journey.

Agile Analytics - A Value-Driven Appr. to BI and Data Warehousing - K. Collier (Pearson, 2012) BBS

Jim was just starting to write the first edition of Agile Project Management when I first met him, and he made book-writing look so easy that I decided to give it a try. As it turns out, its much harder than he makes it look.

My weekly breakfast discussions with Jim were critical in shaping the concepts in this book. He voluntarily served as my developmental editor, reviewing early drafts of sections and chapters and helping me pull things together in a more cohesive and coherent fashion. Jim continues to challenge my assumptions and gives me new ideas and new ways to think about the complexities of development.

He also didnt give up on me when book-writing wasnt my highest priority. Thanks, Jim. Jim introduced me to Luke Hohmann at a time when Luke was looking for somebody with both data warehousing experience and Agile knowledge. Luke is one of the most visionary people Ive ever met.

I was fortunate enough to be the chief architect for one of Lukes innovative ideas: The concepts in this book stem from that experience and have been refined and matured on subsequent projects.

Luke has become a great friend over the past seven years, and I value his wisdom and vision. Thanks, Luke. My team on the aforementioned project remains one of the best Agile teams I have yet experienced either as a participant or as an Agile trainer. Each team member brought a special set of skills and perspectives, and over that first three-plus-year-long project these friends and teammates helped me figure xxxiii.

Ive since had other project opportunities to work with many of these friends, further refining Agile Analytics concepts. These team members deserve much of the credit for validating and tweaking Agile Analytics practices in a complex and real-life situation.

Thanks, guys. Jim Highsmith also introduced me to Scott Ambler along the way. Scott has led the charge in applying Agile to data-centric systems development. Fortunately for all of us, Scott is a prolific writer who freely shares his ideas in his many books and on his ambysoft.

Scott gave much of his time reviewing this book. He has given me much to think about and shared ideas with me that I might otherwise have missed.

Thanks, Scott. I dont think I truly understood what it means for somebody to have the patience of a saint before working with Addison-Wesley editor Chris Guzikowski and editorial assistant Raina Chrobak. As it turns out, I am a painfully slow author who is not very good at applying Agile principles to book-writing deadlines.

Huge thanks go to Raina and Chris, who were amazingly patient as I slipped deadline after deadline. I hope I have future opportunities to redeem myself as an author.

Ralph and I were acquainted at that time and since have become friends and colleagues. I am grateful for his work in this area and for the discussions Ive had with him and the experiences he has shared. Although I have tried not to duplicate what Ralph has already published, I am confident that our approaches are consistent with and complementary to one another. I look forward to future collaborations with Ralph as our ideas mature and evolve.

Finally, the ideas presented in this book have benefited tremendously from smart and thoughtful people willing to review its early drafts and give me guidance. In addition to Scotts and Jims reviews, special thanks go to Jonathon Golden, my go-to guru on project automation, and Israel Gat, expert on Agile leadership and technical debt.

Their contributions were invaluable. Ken has successfully adapted Agile techniques to data warehousing and business intelligence to create the Agile Analytics style. Ken also frequently trains data warehousing and business intelligence teams in Agile Analytics, giving him the opportunity to exercise this approach with various technologies, team dynamics, and industry domains.

In nearly three decades of working in advanced computing and technology, Ken has experienced many of the trends that come and go in our field, as well as the ones that truly transform the state of our practices. With an M. Ken also loves helping organizations adopt and tailor effective approaches and solutions that might not otherwise be apparent.

He combines a deep technical foundation with sound business acumen to help bridge the gaps that often exist between technical and business professionals. Ken has had the privilege of working as a software engineer for a large semiconductor company. He has spent several years as a tenured professor of computer science engineering. He has directed the data warehousing and business intelligence solutions group for a major consulting firm.

And, most recently, he has focused on enabling organizational agility, including Agile software engineering, Agile Analytics, and Agile management and leadership for client companies. It is not a rigid or prescriptive methodology; rather it is a style of building a data warehouse, data marts, business intelligence applications, and analytics applications that focuses on the early and continuous delivery of business value throughout the development lifecycle.

Agile Analytics includes practices for project planning, management, and monitoring; for effective collaboration with your business customers and management stakeholders; and for ensuring technical excellence by the delivery team. This chapter outlines the tenets of Agile Analytics and establishes the foundational principles behind each of the practices and techniques that are introduced in the successive chapters in this book.

Agile is a reserved word when used to describe a development style. It means something very specific. Unfortunately, agile occasionally gets misused as a moniker for processes that are ad hoc, slipshod, and lacking in discipline. Agile relies on discipline and rigor; however, it is not a heavyweight or highly ceremonious process despite the attempts of some methodologists to codify it with those trappings.

Rather, Agile falls somewhere in the middle between just enough structure and just enough flexibility. It has been said that Agile is simple but not easy, describing the fact that it is built on a simple set of sensible values and principles but requires a high degree of discipline and rigor to properly execute.

It is important to accurately understand the minimum set of characteristics that differentiate a true Agile process from those that are too unstructured or too rigid. This chapter is intended to leave you with a clear understanding of those characteristics as well as the underlying values and principles of Agile Analytics.

These are derived directly from the tried and proven foundations established by the Agile software community and are adapted to the nuances of data warehousing and business intelligence development.

Im fascinated by the trials and travails of climbing high mountains like Everest, Annapurna, and others that rise to over 8, meters above sea level. These expeditions are complicated affairs involving challenging planning and logistics, a high degree of risk and uncertainty, a high probability of death for every two climbers who reach the top of Annapurna, another one dies trying!

While it may not be as adventuresome, building complex business intelligence systems is a lot like high-altitude climbing. We face lots of risk and uncertainty, complex planning, difficult decisions in the heat of battle, and the likelihood of death! Okay, maybe not that last part, but you get the analogy. Climbing teams first began successfully conquering these high mountains in the s, 60s, and 70s. In those early days the preferred mountaineering style was known as siege climbing, which had a lot of similarities to a military excursion.

Expeditions were led in an autocratic command-andcontrol fashion, often by someone with more military leadership experience than climbing experience. Climbing teams were supported by the large numbers of porters required to carry massive amounts of gear and supplies to base camp and higher. Mounting a siege-style expedition takes over a year of planning and can take two months or more to execute during the climbing season.

Siege climbing is a yo-yo-like affair in which ropes are fixed higher and higher on the mountain, multiple semipermanent camps are established at various points along the route, and loads of supplies are relayed by porters to those higher camps. Finally, with all this support, a small team of summit climbers launches the final push for the summit on a single day, leaving from the high camp and returning to the same.

Agile Analytics - A Value-Driven Appr. to BI and Data Warehousing - K. Collier (Pearson, 2012) BBS

Brilliant teams have successfully climbed mountains for years in this style, but the expeditions are prohibitively expensive, time-consuming to execute, and fraught with heavyweight procedures and bureaucracy. Traditional business intelligence systems development is a lot like siege climbing. It can result in high-quality, working systems that deliver the desired capabilities. However, these projects are typically expensive, exhibiting a lot of planning, extensive design prior to development, and long development cycles.

Like siege-style expeditions, all of the energy goes into one shot at the summit. If the summit bid fails, it is too time-consuming to return to base camp and regroup for another attempt. In my lifetime and Im not that old. When such projects fail, the typical management response is to cancel the project entirely rather than adjust, adapt, and regroup for another summit attempt. In the s a new mountaineering method called alpine-style emerged, making it feasible for smaller teams to summit these high peaks faster, more cheaply, and with less protocol.

Alpine-style mountaineering still requires substantial planning, a sufficient supporting team, and enough gear and supplies to safely reach the summit. However, instead of spending months preparing the route for the final summit push, alpine-style climbers spend about a week moving the bare essentials up to the higher camps. In this style, if conditions are right, summits can be reached in a mere ten days. Teams of two to three climbers share a single tent and sleeping bag, fewer ropes are needed, and the climbers can travel much lighter and faster.

When conditions are not right, it is feasible for alpine-style mountaineers to return to base camp and wait for conditions to improve to make another summit bid. It is essential that we have a sufficient amount of planning, the necessary support to be successful, and an appropriate amount of protocol.

Our summit is the completion of a high-quality, working business intelligence system that is of high value to its users. As in mountaineering, reaching our summit requires the proper conditions. We need just the right amount of planningbut we must be able to adapt our plan to changing factors and new information. We must prepare for a high degree of risk and uncertaintybut we must be able to nimbly manage and respond as risks unfold.

We need support and involvement from a larger communitybut we seek team self-organization rather than command-and-control leadership. Agile Analytics is a development style rather than a methodology or even a framework. The line between siege-style and alpine-style mountaineering is not precisely defined, and alpine-style expeditions may include some siege-style practices.

Each style is best described in terms of its values and guiding principles. Each alpine-style expedition employs a distinct set of climbing practices that support a common set of values and principles. Im not the first Agile advocate to discuss the analogy between climbing and Agile development. Jim Highsmith made a similar analogy in his book, Adaptive Software Development: Premier mountaineer Ed Viesturs has a formula, or core value, that is his cardinal rule in the big mountains: Getting to the top is optional.

Getting down is mandatory. Viesturs and Roberts I love this core value because it is simple and elegant, and it provides a clear basis for all of Eds decision making when he is on the mountain. In the stress of the climb, or in the midst of an intensely challenging project, we need just such a basis for decision makingour North Star.

In , a group of the most influential application software developers convened in Salt Lake City and formed the Agile Alliance. Through the process of sharing and comparing each of their styles of software development, the Agile Manifesto emerged as a simple and elegant basis for project guidance and decision making. The Agile Manifesto reads: Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.

With due respect to the Agile Alliance, of which I am a member, I have adapted the Agile Manifesto just a bit in order to make it more appropriate to Agile Analytics: Manifesto for Agile Analytics Development We are uncovering better ways of developing data warehousing and business intelligence systems by doing it and helping others do it.

In addition to dealing with large volumes of data, our efforts involve systems integration, customization, and programming. Every activity related to any project either a directly and materially contributes to this primary objective or b does not. Agile Analytics attempts to maximize a-type activities while acknowledging that there are some b-type activities that are still important, such as documenting your enterprise data model.

These include technical, project management, and user collaboration practices. I will demonstrate how you can apply these on your projects, and how you can tailor them to the nuances of your environment. However, the title of this section is What Is Agile Analytics? This is simply a high-level glimpse at the key project traits that are the mark of agility, not an exhaustive list of practices.

Moreover, Agile Analytics is a development style, not a prescriptive methodology that tells you precisely what you must do and how you must do it. The dynamics of each project within each organization require practices that can be tailored appropriately to the environment.

These characteristics simply serve that goal: Iterative, incremental, evolutionary. Foremost, Agile is an iterative, incremental, and evolutionary style of development. We work in short iterations that are generally one to three weeks long, and never more than four weeks.

We build the system in small increments or chunks of user-valued functionality. And we evolve the working system by adapting to frequent user feedback.

Agile development is like driving around in an unfamiliar city; you want to avoid going very far without some validation that you are on the right course. Short iterations with frequent user reviews help ensure that we are never very far off course in our development. Value-driven development.

The goal of each development iteration is the production of user-valued features. While you and I may appreciate the difficulty of complex data architectures, elegant data models, efficient ETL scripts, and so forth, users generally couldnt care less about these things.

Production quality. Each newly developed feature must be fully tested and debugged during the development iteration. Agile development is not about building hollow prototypes; it is about incrementally evolving to the right solution with the best architectural underpinnings.

A user feature is Done when it is of production quality, it is successfully integrated into the evolving system, and developers are proud of their work.

That same feature is Done! Barely sufficient processes. Ive worked on many projects that involved elaborate stage-gate meetings between stages of development such as the transition from requirements analysis to design. These gates are almost always accompanied by a formal document that must be signed off as part of the gating process.

If a data dictionary is deemed important for use by future developers, then perhaps a digital image of a whiteboard table or a simple spreadsheet table will suffice. Since our primary objective is the production of high-quality, high-value, working systems, we must be able to minimize the amount of ceremony required for other activities.

Historically database and data warehouse testing has lacked the rigor, discipline, and automation that have benefited software development efforts www.

Automation, automation, automation. The only way to be truly Agile is to automate as many routine processes as possible. Test automation is perhaps the most critical. If you must test your features and system manually, guess how often youre likely to rerun your tests? Test automation enables you to frequently revalidate that everything is still working as expected. This helps establish continuous confidence that you are never more than a few hours or days away from putting a new version into production.

Agile Analytics teams seek to automate any process that is done more than once. The more you can automate, the more you can focus on developing user features. Too often in traditional projects the development team solely bears the burden of ensuring that timelines are met, complete scope is delivered, budgets are managed, and quality is ensured. Agile business intelligence acknowledges that there is a broader project community that shares responsibility for project success.

The project community includes the subcommunities of users, business owners, stakeholders, executive sponsors, technical experts, project managers, and others. Frequent collaboration between the technical and user communities is critical to success. Daily collaboration within the technical community is also critical. In fact, establishing a collaborative team workspace is an essential ingredient of successful Agile projects. Self-organizing, self-managing teams.

Hire the best people, give them the tools and support they need, then stand aside and allow them to be successful. There is a key shift in the Agile project management style compared to traditional project management. The Agile project managers role is to enable team members to work their magic and to facilitate a high degree of collaboration with users and other members of the project community.

The Agile project team decides how much work it can complete during an iteration, then holds itself accountable to honor those commitments. The Agile style is not a substitute for having the right people on the team.

These principles often become the tiebreaker when difficult trade-off decisions must be made. Similarly, the Agile Alliance has established a set of principles for. We welcome changing requirements, even late in development. Users, stakeholders, and developers must share project ownership and work together daily throughout the project. We value the importance of talented and experienced business intelligence experts.

We give them the environment and support they need and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. A working business intelligence system is the primary measure of progress. We recognize the balance among project scope, schedule, and cost.

The data warehousing team must work at a sustainable pace. Continuous attention to the best data warehousing practices enhances agility. The best architectures, requirements, and designs emerge from selforganizing teams.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. Take a minute to reflect on these principles.

How many of them are present in the projects in your organization? Do they make sense for your organization? Give them another look. Are they realistic principles for your organization? I have found these not only to be commonsense principles, but also to be effective and achievable on real projects. Furthermore, adherence to these principles rather than reliance on a prescriptive and ceremonious process model is very liberating. Agile Analytics is not: A wholesale replacement of traditional practices.

It is important that we keep the practices and methods that work well, improve those that allow room for improvement, and replace those that are problematic. Scrum is perhaps the Agile flavor that has received the most publicity along with XP in recent years. However, it is incorrect to say that Agile was formerly known as eXtreme Programming, as one skeptic told me. In fact, there are many different Agile development flavors that add valuable principles and practices to the broader collective known as Agile development.

Agile Analytics is an adaptation of principles and practices from a variety of these methods to the complexities of data-intensive, analytics-based systems integration efforts like data warehousing and data mart development. Simply iterating. Short, frequent development iterations are an essential cornerstone of Agile development.

Unfortunately, this key practice is commonly misconstrued as the definition of agility. Not long ago I was asked to mentor a development team that had gone Agile but wasnt experiencing the expected benefits of agility. Upon closer inspection I discovered that they were planning in four-week iterations but didnt expect to have any working features until 5. Effectively they had divided the traditional waterfall model into time blocks they called iterations.

They completely missed the point. The aim of iterative development is to demonstrate working features and to obtain frequent feedback from the user community. This means that every iteration must result in demonstrable working software.

For systems integration; its only for programming. This complexity calls for an approach that supports a high degree of customer collaboration, frequent delivery of working software, and frequent feedbackaha, an Agile approach! An excuse for ad hoc behavior. Some have mistaken the tenets of Agile development for abandonment of rigor, quality, or structure, in other words, hacking.

This misperception could not be farther from the truth. Agility is a focus on the frequent delivery of highvalue, production-quality, working software to the user community with the goal of continuously adapting to user feedback. This means that automated testing and quality assurance are critical components of all iterative development activities. We dont build prototypes; we build working features and then mature those features in response to user input. Others mistake the Agile Manifesto as disdain of documentation, which is also incorrect.

The keyword here is sufficient. Sufficiency implies that there is a legitimate purpose for the document, and when that purpose is served, there is no need for additional documentation. This is a natural inclination for new Agile practitioners, and I will provide some. However, I need to reemphasize that Agile Analytics is a style, not a methodology and not a framework. Figuratively, you can absorb agility into your DNA with enough focus, practice, and discipline.

Youll know youve reached that point when you begin applying Agile principles to everything you do such as buying a new car, remodeling a bathroom, or writing a book. This is not a substitute for any of the more comprehensive technical books on data warehousing but should be sufficient as a baseline for the remainder of the book. Data Warehousing Conceptual Architectures Figure 1.

This is a high-level conceptual architecture containing multiple layers, each of which includes a complex integration of commercial technologies, data modeling and manipulation, and some custom code. The data warehouse architecture includes one or more operational source systems from which data is extracted, transformed, and loaded into the data Source Systems Intergration Presentation Analysis. These systems are optimized for the daily transactional processing required to run the business operations.

Data from these sources is extracted into an integration tier in the architecture that acts as a holding pen where data can be merged, manipulated, transformed, cleansed, and validated without placing an undue burden on the operational systems.

This tier may include an operational data store or an enterprise information integration EII repository that acts as a system of record for all relevant operational data. The integration database is typically based on a relational data model and may have multiple subcomponents, including pre-staging, staging, and an integration repository, each serving a different purpose relating to the consolidation and preprocessing of data from disparate source systems.

Data is extracted from the staging database, transformed, and loaded into a presentation tier in the architecture that contains appropriate structures for optimized multidimensional and analytical queries. This system is designed to support the data slicing and dicing that define the power of a data warehouse.

There are a variety of alternatives for the implementation of the presentation database, including normalized relational schemas and denormalized schemas like star, snowflake, and even starflake.

Moreover, the presentation tier may include a single enterprise data warehouse or a collection of subject-specific data marts. Some architectures include a hybrid of both of these. Presentation repositories are typically implemented in the same technologies as the integration database. Finally, data is presented to the business users at the analysis tier in the architecture. This conceptual layer in the system represents the variety of applications and tools that provide users with access to the data, including report writers, ad hoc querying, online analytical processing OLAP , data visualization, data mining, and statistical analysis.

BI tool vendors such as Pentaho, Cognos, MicroStrategy, Business Objects, Microsoft, Oracle, IBM, and others produce commercial products that enable data from the presentation database to be aggregated and presented within user applications.

This is a generalized architecture, and actual implementations vary in the details. One major variation on the Kimball architecture is the Inmon architecture Inmon , which inserts a layer of subject-specific data marts that contain subsets of the data from the main warehouse.

Each data mart supports only the specific end-user applications that are relevant to the business subject area for which that mart was designed.A project plan that focused on adapting to changes rather than meeting a set of frozen contractual requirements would have greatly improved user satisfaction with the end product.

Ultimately, project costs far exceeded the budget, users were unsatisfied, and the project was placed on hold until further planning could be done to justify continued development. These gates are almost always accompanied by a formal document that must be signed off as part of the gating process. What does your team need to do to keep its viewer ratings high every single iteration? Learn more about Amazon Prime.

Agile Software Development Series Paperback: Explore Phase Waterfall-type models include some variation of these phases: Foremost, my good friend Jim Highsmith has been my trusted adviser and mentor since the beginning of my Agile journey. Agility is a focus on the frequent delivery of highvalue, production-quality, working software to the user community with the goal of continuously adapting to user feedback. We need just the right amount of planningbut we must be able to adapt our plan to changing factors and new information.

ALYSSA from Ann Arbor
I enjoy sharing PDF docs neatly. See my other articles. I am highly influenced by production car racing.